AI animation: "a layer of window paper" is still close to radical change

Original source: East-West Entertainment

Author: Xia Qingyi

Image source: Generated by Unbounded AI

At a time when generative AI technology is leading industrial innovation, animation is one of the first industries to pay attention to and be affected.

Animation has always been a technology-driven industry. The development of computer graphics has triggered tremendous changes in the form and expressiveness of animation itself, market and industrial structure; The current core production capacity and business model problems in the animation industry are still constrained by technical forces.

The animation industry expects new changes, which must be caused by technology. Now it seems that this technology is AI.

AI is nothing new to animation, especially three-dimensional animation. But in the past year, from ChatGPT, Stable diffusion to Midjourney, it has subverted many people's traditional perception of AI.

Many animation companies and platforms are realizing that AI can be used as a tool for commercial use, to meet more individual requirements, and to develop capabilities for their own content orientation. This also makes animation companies with different positioning and business models more consideration of the use of AI.

the "daily broadcast" model that was once unimaginable due to capacity constraints; Or as a part of the IP industry chain, take the initiative to lead the rhythm of IP development with more economical investment; Or establish a more general process and library assets to open up the production from animation to games... All may become a reality with the intervention of AI.

When animators are freed from tedious physical and assembly line work, the industry will be more driven by creative talents and explore new artistic paradigms.

This is a huge opportunity.

Of course, the opportunities ahead correspond to the challenges of the present. Change and innovation often come with trade-offs and adaptation.

The animation industry wants to seize the opportunity, but the timing and posture are all problems. A reasonable model between AI technology capabilities, application cost input and return has not yet been established, and there are still many uncertainties. In the future, with the development of technology, animation companies and platforms as application parties will also have decision-making.

But no matter what, the general direction is always embracing AI.

Rationally, not every iteration of technology will revolutionize an industry. When you are still in the process of change, whether it is really used and to what extent it is not the most important thing, it is only to understand the environment and trends and make correct judgments.

AI brings opportunities for animation companies, and failure to seize them can also become a crisis. After all, technology is moving too fast. If one day AI becomes very useful, no one wants to be left behind at that time.

Most significant cost reduction and efficiency increase

As early as 2021, OpenAI released DALL· E, based on the text token to generate an "avocado-shaped armchair" out of the circle, has made many animators pay attention to AI. However, due to factors such as technology maturity, corresponding talents and unclear application scope, it mainly stays in using AI to do some creative exploration.

In the past year, the rapid iteration of Stable diffusion and Midjourney has not only opened up the cognition of AI on the C-end, but also directly promoted the mass application of AI in many industries, including animation-based on the open source architecture of the former, the development of plug-in tools suitable for their own business; The latter's drawing ability has been continuously upgraded, especially in the front-end creativity and planning link, the application value has almost won the industry consensus.

In August this year, Tencent Video released a trailer for "Magic Game: Artificial Intelligence Assisted Chapter", claiming to be "the first artificial intelligence-assisted animation in China". As early as around the Spring Festival in 2023, Tencent Video deliberately tried to apply AI to animation production, and finally hit it off with Red Dragon Pictures.

In fact, animation has become a relatively stable commercialization track for Tencent Video. In the case of relatively locked costs in the entire animation industry, the core of this business logic is to use animation content to stably attract new users, and the higher the popularity of animation, the greater the revenue. Popularity depends not only on the quality of the content, but also on the frequency of updates.

As a producer, Ding Yu, founder of Red Dragon Pictures, said that at that time, both parties believed that there was value in exploring the use of AI to make animation, whether from the perspective of reducing costs and increasing efficiency or finding new styles, "After talking in March, the implementation began in May."

Cost reduction and efficiency improvement are the primary starting point for almost all animation companies to apply AI at this stage.

Compared with two-dimensional animation, the overall creative process of three-dimensional animation greatly depends on the digital platform, and AI technology has deeper room for intervention. Whether it is two-dimensional or three-dimensional animation, the early stages of the art production process are similar, including character and scene development, concept art, original art, and so on. These links are the deepest part of AI application at present, and the cost reduction and efficiency increase are particularly significant.

For example, character creation, the founder of ASK animation, which is mainly two-dimensional animation, said that in the past, the writing and drawing stage may only take a few hours, but before it took months to think, now using AI to help coordinate ideas, the time to find inspiration can be greatly shortened.

The same goes for 3D animation. Zhang Xin, technical director of Black Rock Network, revealed that the newly launched animation project this year adopts more AIGC processes, and the early art part only takes 1/3 or less of the estimated time than the project "Yujia Lingyunzhi" developed last year.

3D animation "Imperial Armor Lingyunzhi"

This is because the time for the director team to design a concept is relatively short, but if these concepts are presented to the director team with better visual effects using the traditional art process, it takes a relatively long time. And different projects have different styles and different requirements for personnel skill levels, resulting in a high dependence on art talents.

With AI assistance, pure creativity and planning art are still completed by people, but most of the detailed tasks are handed over to AI, which is equivalent to people doing preliminary design, AI completing "manual work", the demand for manpower and time is much reduced, greatly reducing the cost of trial and error. This makes the work that was originally a team of less than 20 people that needed to be completed in a cycle of 8-10 months into less than 10 people and can be completed in about 3 months.

"It has also changed the requirements for team configuration, and in the past, it was necessary to have a few very senior people to lead an art team of less than 20 people to complete the work, but now only two senior artists, with a small number of junior and intermediate personnel, can reach a level that was difficult to achieve before," Zhang Xin said.

Ding Yu also said that in the past, to make character setting drawings, you need to find references and team communication and discussion, and then art to produce a line draft, finalize the color draft according to the line draft, and step by step may take a week to produce a complete colored character. After using AI, first summarize the keywords that everyone thinks, and the AI trainer will draw a picture based on the keywords. With the accumulation of training, AI's understanding of keywords becomes more and more accurate, and its efficiency continues to improve.

For example, using AI tools, 40 images can be produced in one night. Fine-tune keywords based on 40 images, such as more specific details such as 'American yellow T-shirt'. After getting 40 images, select the satisfactory parts on each image, and then refine and re-summarize the key words, and finally generate a new image.

On the one hand, this approach reduces the cost of communication, "shortening the whole time by 1/2 or even 2/3"; On the other hand, the use of AI drawing also gives design and development more choices, selecting the favorite part from various generated pictures and re-drawing to improve the final effect, and finally achieve an unattainable effect in a shorter time.

In addition to the direct improvement of animation idea incubation and production capacity, AI also has a clearer application direction in animation IP development.

Xu Bo, founder of Molecular Interaction, who launched animations such as "Inhuman" and "Youbeast Yan", said that the company's core business and positioning are different, and the value of AI is also different. "We are positioned as an IP-type company, and animation is a content form and a link in the entire IP, so we will not only look at AI from the perspective of animation, but also think about the value of AI to comics and IP operations."

For example, at the animation level, it currently attaches importance to the assistance of AI to key original paintings and scenes, while in IP derivative developers, in addition to AI to help follow-up material design, derivative development, etc., there are also AIGC generated posters, AI coser and other directions, which can play a good role in marketing.

"Inhuman" IP

The real "overtaking" opportunity is to subvert the whole process with AI

In general, animators generally see AI as a new opportunity. Just like when Ding Yu and Tencent Video cooperated, I felt that this might become an opportunity to overtake on a corner. Because the current AI is not only used as a tool to assist a certain link, the development of the entire industry and the whole process may be adapted to AI as a whole. This possible disruption is worth trying.

In other words, the industry's expectations for AI do not stop at AI intervention in pre-planning and other links to help reduce costs and increase efficiency.

But higher expectations are not easy to meet. Even if it is "reducing costs and increasing efficiency", the animation industry has generally felt a certain "bottleneck" after this year's application development.

Among them, the degree of technological development is the core factor, and it is not only technical issues.

As a platform that provides AIGC painting tools and AI creation services, Unbounded AI has been cutting the ACGN field. In the view of Zhao Jiecheng, head of the Unbounded AI Content Center, animation is the most difficult to cut into.

Comics have less technical requirements, no need to model, make up intermediate actions, etc., AI-assisted creativity, line drawing, coloring, etc. have almost completed most of the work, and manual more investment in storytelling, storyboarding, etc., "in March this year, it can only complete 30%-50%, and now it can assist 70% of the work." For the comic platform, this is equivalent to solving many core problems and directly fighting for more space for commercialization.

The original drawing line and production line of many domestic game companies are usually separated, which is convenient for process division. Zhao Jiecheng said that in order to reduce the cost of project establishment, many games no longer invest a lot of cost and energy to do too much setting in the early stage, and the emergence of AI has allowed game companies to significantly cut the outsourcing of original art, even reaching 70%-90%. Because some games themselves do not directly present the original art, mainly based on the original art modeling, these reference modeling materials are easier to be replaced by AI.

Unbounded AI generates images

In contrast, animations are more demanding. Animation needs to make a direct and coherent picture, and AI has problems in coherence, light processing, etc., so from the perspective of the entire process, AI's involvement in animation, especially three-dimensional animation, is not deep, unless the overall replacement production process, such as replacing it frame by frame with AI after real shooting.

"But now there are no animation companies in China that use frame-by-frame animation to do animation, and generally have their own fixed production process. This leads to more AI being able to intervene in terms of pre-setting, scenes, tasks, coloring, etc.," said Zhao.

In fact, different animation companies in China, and even different projects of the same company, may have different production processes. Not only are third-party companies such as Unbounded AI difficult to provide matching standardized solutions, but animation companies will also face similar difficulties in finding AI tools and conducting internal technology research and development.

Including the widely adopted Stable diffusion and Midjourney, various technical tools have not yet developed to the extent that they are customized and developed specifically for the animation industry, resulting in animation companies basically only passively adapting to the development of AI technology.

In addition, most small animation companies cannot afford to invest in the development of plug-ins that suit them. For large animation companies and even platforms, it is not easy to invest in research and development at this stage and further change the whole process of animation production.

First, the cost and direction of technology input are not clear enough, there is a lack of relevant talents, and there is no mature business model to recover costs at a certain time; Second, there are still many processes that use AI, which may be more expensive or less efficient than manual labor.

Tencent Video's AI animation project has also undergone heated discussions in the early stage. "As a forward-looking layout, the "Magic Game" project actually tests the strategic determination of the platform."

"Magic Warrior Artificial Intelligence Auxiliary Chapter"

For many animation companies, such as 2D animation, one of the most important cost issues is frame fusion between frames, how to move from one action to another.

Yu believes that two-dimensional animation is still difficult to achieve a state of de-artificialization and is constrained by hard costs. Although early attention has been paid to technologies such as AI simulation style and filling in intermediate frames, cutting-edge research and development results are still difficult to apply to the industry, and the final cost may be much higher than human generation. "This creates a paradox."

The cost of 3D animation is hand. Purely by hand, a team of 200 people may need to do it for a whole year. And "hand K" is not a pure technical assembly work, can reflect the animator's personal sense of art, rhythm, and some of the exaggerated actions and effects beyond the reality of motion capture can only be completed by hand K, which is more common in animation.

Yang Lei, vice president of Ruosen Digital, said that the "Bad People" series has not yet adopted AI, and it is estimated that AI applications of such a quality level of animation account for up to 20%, and the rest still require a large number of manual teams.

On the one hand, at present, the ceiling of three-dimensional film-level animation is in hardware, such as solving is a very time-consuming link in three-dimensional animation production, which must be realized by computing power; On the other hand, animation requires too much for art integration, for example, it is currently difficult for AI to do a good job of storyboarding, and even if the technology is achieved, it may not have good artistic effects. "UE5's Metahuman already has a surface capture system that can directly replace the model, but it still doesn't meet the requirements for high-quality film-level content, and it still requires some work by hand."

3D animation "Bad People Who Draw Rivers and Lakes"

AI problem, also "human" problem

AI has not yet been able to replace human labor in some links, but it has caused many concerns among animators.

Many ordinary practitioners, from the end of last year worried about whether AI will replace their own anxiety, gradually evolved into what kind of capabilities are needed in the AI era. As a result, the animation industry's pricing standards have been volatile. Large integrated animation companies benefit from AI, but some small teams with single skill categories and business models are vulnerable to AI impacting personnel structure and service capabilities, and the risk is higher in the current direction and environment.

Yu believes that the most vulnerable positions at the moment are the concept link, while planning companies and creative talents are relatively stable. Of course, in the long run, AI will eventually bring systemic changes that could eventually eliminate more than a third of the entire process. There will be fewer assistant jobs, and the core will be concentrated in a few hands.

ASK Animation Project Works

This puts forward completely different requirements for animation talent skill points - when people are liberated from assembly line work, talents who know what they need and what the market needs are more rare. In the eyes of many animation companies, such talents have real creativity than superficial professional skills. Animation productivity will shift from executive talent to creative talent driven.

For example, the original excellent art skills and color relationships can now be better replaced by AI; Originally drawing a scene map in 3 hours, it may only take a few minutes for AI to draw a picture. But this requires practitioners to have a good aesthetic, sort out their own logic and methodology, and be able to judge what kind of scene is beautiful, more in line with the project style, and more advanced in the original preset environment.

Under this trend, Xu Bo believes that new career segments will emerge in the future, including trainers for AI painting. "There are not many such talents at present, and you need to understand how to do this work without AI, have a fairly good art foundation and aesthetics, and learn to use AI in various ways for training and description." This will be a valuable skill for some time to come. ”

For animation companies, it is naturally not easy to find the right AI talent at this stage.

Zhang Xin said that Black Rock Network noticed AI in 2021, and at that time wanted to look for computer graphics or confrontation generative network related talents, but found that there was a basic gap in China, and no one was found in recruitment for nearly a year, so it had to shelve research and development in this direction. Until the second half of 2022, Stable Diffusion has set off a storm, and it is still difficult to recruit relevant talents. Whether this technology can be successfully realized as an alternative to localization is not optimistic.

It is difficult to recruit the right people, in addition to the pioneer nature of this technical direction itself, from the perspective of the animation industry, first of all, animation practitioners are passive acceptance of most of the technology, and it is difficult to make breakthrough changes themselves; Secondly, in terms of personnel training, many practitioners are still the mentality of artists, but the most important thing in commercial animation products is content and efficiency, and in the case of no problem with content expression, whether the picture is drawn by the artist one by one or calculated by the computer is not the most important thing for the audience.

"It's a bit cruel to say, but many people's mentality is difficult to change, and they are resistant to new technologies. Now practitioners' views on AI are similar to our previous generation of old-school painters looking at digital painting. Zhang Xin said.

If you can't recruit ready-made talents in the market, you can only cultivate them from internal teams. The changes in the training business of animation companies also reflect the changes in the skill pursuit of domestic animation talents - from mainly teaching traditional digital painting in the early days, to gradually adding real-time rendering courses, and this year an AI section was added to guide students to use AI to assist creation.

Optimism or pessimism, the future is here

Faced with these problems, animation companies have not stopped paying attention to AI. The outbreak of AI in the past year has made many people believe that leapfrog development is not too far away, perhaps "a layer of window paper".

Ding Yu said that although there was a lot of preparation in advance, it was actually quite unexpected that the decision to use AI to do animation could pass, because in the past two years, major platforms have generally taken cost reduction and efficiency improvement as the keynote, and the decision to use AI to make animation is not easy. However, the AI technology and process applicable to local animation companies can only be realized in the accumulation of one project after another, and once the training of AI is stopped, there will be no further progress.

The current development of technology is bringing some new possibilities.

Yang Lei has been paying attention to the technical tool of Wonder Studio, because the direction of the product is related to the direct process and cost change of animation - after the live action, the finished assets can be directly replaced, and a series of work such as binding can be automatically completed. Although it does not yet include functions such as costume solving, the effects of the program have the opportunity to subvert the existing animation production process and realize true film-level applications.

As the tools in this direction become more mature, the hand-to-K ratio may drop to just 10%-20%. Effects that previously required advanced animators may only require beginner ones in the future, and the number of people will be greatly reduced.

In addition, in the past, lighting and other effects needed to be rendered, but now real-time lighting can be adjusted through parameters, what you see is what you get, and the film is directly released, which is very fast. "Maybe one day in the future we can do daily animation," Yang said.

At present, technology giants such as NVIDIA are exploring the direction of "what you see is what you get", which may also lead to dramatic changes in the animation industry's processes and structures.

Zhang Xin believes that the three-dimensional animation industrial process is through more than a dozen processes such as models, animations, lighting, special effects, and synthesis, and finally presented to the screen. The final form of AI must skip all intermediate links, do not need the above processes, and save a lot of costs.

"Now AI is still assisting creators to generate preliminary images and ideas. I believe that in the future, AI will not only assist in accelerating in a single process such as modeling and rendering, but also directly destroy the current animation production process and give the final rendering result. As long as the computing power is enough, what you see is what you get is what you get when AI tools produce it, which will be a sea change. ”

In the process, animation companies can also accumulate future-oriented competitiveness. This includes training your own model library in advance and establishing a process that suits your own style based on director needs, project cycle, quality, and cost.

The richness, personalization style and copyright of model training are all directly related to the material library. Yu said that training based on stylized materials within the company may solve the copyright problem; At the same time, it can not only continue the company's creative style, but also may produce results that exceed expectations, and recreate along new results, and the ideas and directions will become broader.

By continuously training models that match the needs of companies based on their own materials, these assets are also expected to become the core competitiveness of animation companies and expand the boundaries of animation companies' business models. For example, in addition to the animation business, Black Rock Network is also developing its own games, and shortens the early release cycle by applying new AI processes and model libraries to game development, hoping to realize that the next animation and game will be launched at the same node.

From the perspective of the entire animation industry, in addition to animation companies, individuals may also change - AI technology makes animation more and more dependent on personal talent and style, various AI tools appear, reduce the threshold of animation production, animation UGC or will be more normal, which may also bring opportunities for animated short videos.