How was it using Devin? ～Case Studies and Key Points for Introduction
I did a roundtable with Devin on Findy yesterday, what was the response?
Yesterday, February 17, 2025, Findy's Devin Roundtable "How was it using Devin? - Case Studies of Use and Points to Consider When Implementing", the response confirms some of the impressions from the postings on X. Specifically, participants and stakeholders shared their value and impressions of the event, with the following points mentioned
- It is reported that nearly 1,000 people have signed up to participate, indicating the high level of attention to the event.
- Participants said it was "very good," and specifically, the following insights were popular: "the perspective of considering Devin as a new member," "an environment where AI is easy to work in is also friendly to new employees," "how to use it by asking it to be small and see how it goes," and "using it only in private repositories.
- Some participants also commented that it was particularly meaningful to be able to verbalize the division of roles, "0→1 is done by humans, 1→10 is done by Devin.
- The speakers and organizers also expressed their satisfaction with the free discussion and their willingness to further develop the event, including the release of a $200 invitation link. Overall, the responses on X give the impression that the event provided practical insights to participants and was received favorably. However, these are only the opinions of a few people who posted on X, and more extensive feedback may be needed to make an exhaustive assessment of the overall response. Nevertheless, the fact that the positive feedback was noticeably positive suggests that the roundtable discussion was a certain success.
2. Devin's behavior of uploading images to an external server without permission
.
- An example of a request to Devin for a screen capture, which was uploaded to a mysterious (not originally intended) external server.
- Basic authentication was in place, but it was a very close call in terms of security for the company.
- From then on, the rule was strictly "put the screenshot directly on the slack.
- Because Devin is free to use shells and browsers, there is a possibility of registering and using extraneous external services.
- When used in a company, it is necessary to decide in advance "how much freedom to give Devin" and manage the risk.
3. "40,000 yen was melted down in a long session" case
.
- The task request to Devin was so large that the plan collapsed, and after endless work and sleep, it finally resulted in a pay-as-you-go charge worth 40,000 yen...an episode.
- Devin (the current version) behaves more and more suspiciously when large tasks are assigned all at once, and is prone to a sharp increase in credit consumption (ACUs).
- If large tasks are requested all at once, there is a great deal of failure and waste.
- In practice, the tips are to "divide tasks into small pieces and request them" and "divide them into units that can be completed in about 10 to 20 ACUs.
- Due to the specifications of AI agents, continuing to run them with messy instructions tends to result in painful billing and time wasting.
4. Non-engineers such as designers and PMs participate in code editing
.
- Devin works in the cloud and can be directed via slack.
- Therefore, the report introduces a case in which even those who cannot maintain a local development environment (e.g., designers and PMs) can request wording corrections and light logic modifications by the designers themselves, and proceed directly to release.
- "Democratization of code" will help to break away from a situation where change tasks are concentrated on only a small group of engineers.
- The speed of product improvement and the degree of freedom to reflect ideas will be increased.
5. Devin as "engineer to scale out"
- Editor-connected AI (e.g., Cursor) is a means of "scaling up" to "enhance a single human engineer.
- In contrast, Devin pointed out that "it works like another engineering employee has been added" = "scaled out" kind of presence.
- When the workload increases, it is possible to use "start multiple Devins in parallel → proceed with development all at once, and reduce the number of Devins when you are free.
- Instantaneous resource increases and decreases, which are difficult to achieve with human employment, can be realized.
The key points are "division of tasks" and "use of the right tool for the right job.
- Devin is basic: "The longer you remember within one session, the more likely you are to become gradually confused.
- It works better to intersperse detailed checkpoints, such as dividing tasks that are likely to take more than 10 ACUs into smaller pieces and making pull requests each time.
- Combined with ChatGPT and other LLM tools, the flow of "first summarize the fluffy requirements into a rough design and requirements → ask Devin to implement" is becoming established.
7. Difficulty of know-how management (knowledge) with "IF-ELSE all over the place"
.
- Devin's Knowledge function is useful, but if it is divided too finely, it is easy to make "mistakes by omitting references.
- On the other hand, it is difficult to update a whole large knowledge base in one place.
- In the end, it is safe to "document the rules that are really necessary in the repository (e.g., summarize them in README) and put them in a form that humans can also understand.
8. Log all "processes and thoughts" that Devin worked on
.
- Devin executes the work plan step by step on the container, and the plan, shell operation history, and editor operations can be viewed later.
- It is like being able to follow a detailed log of "what the newcomer thought and how he/she worked," which can be a learning material for the entire team.
- However, be careful about the scope of disclosure (confidential information may be inadvertently included).
--
- Summary
While actual use of Devin can lead to unexpected security risks and major billing problems,
- Non-engineers can also participate in development
- Outstanding productivity gains depending on task management and division
- Advantages of having all logs visible
New possibilities that do not exist in conventional development sites are increasingly being realized, such as [Stricter operational rules (be careful with confidential information and public repositories, make tasks more detailed, organize where to put knowledge...) The most important point to be gained from the speakers' real-life experiences is that the development culture can evolve significantly by putting in place
2/17 18:30~
overtime https://www.youtube.com/live/ED2rPmRuA60
--
1. Confusion due to session use
.
- Once up and running, Devin continues the task, memorizing the conversation history in the thread.
- Mixing different tasks in the same session will result in a mess of instructions, confusion, or even more trouble than 10 ACUs.
- There is a pattern of "if you ask for Task B before Task A is finished, the work will be scattered as Task B is started without a pull request for Task A."
- Conclusion: "one task, one session" is a safe bet.
--
2. Problem with review volume spiking
.
- Devin's ability to scale out (multiple tasks can proceed simultaneously) can easily lead to a "human dies in review" situation.
- When the pull requests are huge, it's hard to review them after all, and it's hard not to break them up into smaller tasks every so often."
- Automated testing will lower the cost of operation verification, but still how to streamline the review process is a future issue.
--
3. All team members use Devin or squeeze
.
- Ubie example: Including designers and POs, everyone is free to use it → Facilitates unexpected improvement ideas and democratization of the code.
- Helpfeel's example: "Initially, the introduction was focused on managerial level and limited members due to billing and information leakage risk management.
- Both have their merits and demerits, and the degree of rulemaking is crucial when "convincing supervisors and security personnel".
--
4. Advantages of having all work logs in text
.
- Devin keeps a history of all planning steps, shell operations, and editor operations.
- This is like visualizing the "know-how in the mind of a newcomer" and is a learning resource for the team.
--
5. "Non-engineers can touch the code" because no local environment is required.
- In addition to the first half of the presentation, he also talked about cases where designers and PMs were able to complete minor wording and layout modifications on their own.
- Since there is no need to set up a local environment in the first place, "just leave it to Devin from Slack" to create a branch and submit a pull request.
- Result: Faster detailed modifications, less work for engineers → Engineers focus on more difficult logic parts.
--
6. Misconception of Cost Savings with Sleep
.
- Doesn't it cost a lot of money to leave Devin up and running? → In fact, "the cost of being on standby is very small.
- The main credit is consumed by "querying the LLM = a large number of thought steps", so the more you keep shaking up large tasks, the higher the cost.
- It's harder for people to worry too much and be driven by the need to put Devin to sleep as quickly as possible."
- Conclusion: Sleep is necessary, but you don't have to be that nervous.
--
7. "Will we no longer need engineers?" Issue
- Will engineering jobs become less interesting as AI agents like Devin become more sophisticated?" The question is.
- The common view of the three: on the contrary, engineers can do much more and it becomes more interesting.
- AI can take care of the "itchy parts" and engineers can spend their time on more upstream and creative work.
- On the other hand, those whose identity was "the joy of writing code" may experience an identity crisis.
- In reality, however, it is possible to "deliver even greater value to society" by becoming an AI master.
- → Rather, there is more room for skill.
--
8. The parts of the job I don't enjoy are disappearing
.
- Some see Devin's arrival as "eliminating the need to do miscellaneous and repetitive tasks that you don't want to do.
- With the right approach to requests, many tedious areas can be delegated, and people can concentrate on what they like to do and on tasks that allow them to be creative.
- The result: "work that engineers don't want to do but have no choice but to do" will be drastically reduced, and even working hours may be shortened.
--
9. Disparity between those who use it vs. those who don't
.
- Some are concerned that Devin and others who cannot handle AI agents will be left behind.
- However, you don't have to be an engineer to give natural language instructions in Slack → rather low threshold.
- Furthermore, there should be "knowledge bridge" and "usage support" jobs to support those who are not comfortable with AI.
- Becoming familiar with AI agents early on will certainly increase their value to the organization.
--
- Summary In the second half, the discussion went more into operational design and [* identity as an engineer. 1.### Confusion and cost explosion due to mixed sessions
. 2.### Reduce review stress by "dividing tasks into smaller pieces + quick pull requests + emphasis on automated testing" . 3.### Use all or only limited members → Different companies have different strategies 4. ### detailed work log visualization is a huge benefit and risk 5.### Not that engineers will become unnecessary, but more upstream and creatively interesting . 6.### A world where one does not have to do miscellaneous tasks is an ideal world for "those who don't want to work" .
[The engineers' goal is to go beyond the "work" that Devin does, and to use their creativity to take on bigger challenges. It was impressive to see how those who actually tried it enjoyed it, saying, "It lowered the hurdle for me to try big things that I couldn't do before.
Announcement Slide
nishio I realized 30 minutes ago that there was a slot for an announcement slide in the slides for today's event, so I hurriedly made the content, but @teramotodaiki probably hasn't noticed yet. Maybe @teramotodaiki hasn't noticed yet. teramotodaiki mjsk
- Advances in AI and Plurality
- Japanese translation of PLURALITY by Audrey Tan and Glenn Weil | Cybozu Shiki Books
- Funding the Commons Tokyo 2024
- Feeding on conflicting opinions, we will use digital technology to re-tangle the world's divisions. Unraveling the New Concept of Plurality──Audrey Tan x Glenn Weil x Haruyuki Seki, Code for Japan | Cybozu Style
- AI Engineer Takahiro Yasuno and Cybozu's Yoshihisa Aono If we change our "sense of distance" from technology, we may be able to create a society where no one is left behind | Cybozu Style https://cybozushiki.cybozu.co.jp/articles/m006208. html]
X/Twitter response
.
tonkotsuboy_com Thank you very much for the Findy Devin event, it was meaningful to talk freely with Mr. Teramoto and Mr. Nishio without rules. It was especially good that we could verbalize "0→1 is done by humans, 1→10 is left to Devin".
chroju To take advantage of AI, documentation, language of code comments, and process automation like CI will be really important.
integrated1453 It's the idea of Institutional Memory where the entire organization keeps a record of what it has learned from past experiences. I'd like to have everything accumulated only in Devin's Knowledge to be vomited into documents as appropriate.
- Yes, the interesting thing about Devin's thoughts and knowledge being textualized and accumulated, unlike human thoughts and knowledge, is that they do not accumulate within the individual but within the organization!
integrated1453 I know that as an organization, we would like to develop a natural language textualization of custom instructions to AI agents so that they are not locked in by Devin and can be used by Cursor I understand that this is not yet a trend, so I would like to see it grow so that it can be used by Cursor, GitHub Copilot Agent, and others. I don't think the trend is there yet.
potix2 It's amazing the feeling of tacit knowledge becoming visible when working with Devin.
potix2 I'm sure there will be a knowledge optimization feature or something like that eventually.
- If your style is to make one big document like Mr. Teramoto's, you can copy it and show it to ChatGPT for discussion.
katsuhisa__ Interesting reference to the granularity of Devin's Knowledge. It seems to be a good idea to adopt the concept of creating one big knowledge as much as possible.
- Unlike the ChatGPT itself, Devin's memories can be edited.
dai___you What's in our heads is hard to access, but devin-kun leaves it in text, so it's highly searchable, which has a lot of potential.
Dakuon_Findy So there is a land line to the issue of MTG not keeping minutes...
sjisjis Today's golden rule, it fits. Scale up Cursor and others Devin scales out - Scale-up and scale-out
Dakuon_Findy Ubie has a system that Devin can use for everyone who raises their hand, and Helpfeel has the authority to outsource to outsourced workers. Once narrowed down to Mgr class only. js4000all "There are two ways to deal with devin, one is to limit the number of people responsible for assigning tasks, and the other is to allow anyone to use it. The former may be the easiest to introduce.
Personally, I think "anyone who wants to use it can use it" is the ideal for the future.
but since the cost is high at this point, I felt it would be a realistic step to focus on the manager class first.
- It's easier to think about the value to the company when you have more people in the company who have experienced it.
Anyone who wants to use it is a member of the group.
The expression "more people can tinker with source code" and "democratization of code."
imamura_ko_0314 Designers (non-engineers) can now modify simple wording/screen corrections they notice themselves.
In some cases at Ubie, some code is developed by designers using devin. --- Aiming to be a Generative AI Native Company - What we are doing at Ubie｜masa_kazama
You can start by giving instructions in Japanese, even if you don't know what to do in the source code or how to submit a Pull Request.
Narrowing it down to managers
freddiefujiwara It is interesting that you limit devin to the managerial level who are properly responsible for costs. Positioning it as a variant of outsourcing.
- Ah, I see, "a sense of responsibility for costs".
chroju Devin, I started using it last week, but I certainly get the feeling that it would be quite suitable for managers to leave it to me to do what they want me to do single-handedly.
yumion7488 It's easy to implement for a team because there's only a limit on the number of executions per month (1 ACU = about 15 minutes of uptime, 250 ACUs per month), not per user. smasato I think the fact that they don't charge by the number of accounts, but by the amount of ACUs consumed, is another factor that makes it easier for all kinds of people to touch the system. smasato I'm thinking that using one Devin on multiple GitHub Orgs would not fit the Devin philosophy. I understand that the idea is to gradually grow a machine of engineers called Devin working in one > organization, and if it is to be used in multiple organizations, it would work better to separate the Devin itself.
hisayuki_mori Devin's Strengths Employees to whom the Labor Standards Act does not apply freddiefujiwara devin not retiring
freddiefujiwara I think the stock prices of Offshore and SES companies will drop over the next few years.
nishio This is something I didn't mention in the delivery, but compared to my experience of crowdsourcing and offshoring overseas, I think "the cost of hiring which people to choose has been reduced to zero. I think it's revolutionary to be able to start any number of people in parallel, depending on the workload.
edgissa GAFA or something like that, already utilizing Agents for internal use for over a year.
nishio I didn't have a chance to mention it in the delivery, but Cybozu has been operating an AI that will act as "one of our employees" on the groupware that employees use for their regular work since December 2023. We have been operating an agent, for more than a year.
When you use Devin, you say, "I see! You do it this way!" (I have been able to better verbalize the "future of working with AI").
smasato Devin history is not shared between sessions hisayuki_mori I can't do the previous conversation, but I could do it after seeing a PR that was done in another session.
kgsi I guess you can afford 100 or 200 requests. erm1116_ I feel that it would be very nice to be able to proceed with tasks asynchronously and in parallel. After all, there's going to be some sort of bottleneck, like the person doing the PR review.
Dakuon_Findy It was a very interesting experience to be thrown from your phone! integrated1453 I give work orders to Devin on Slack on my phone while I take care of the kids and lie on the couch! I give Devin work orders while taking a walk and taking a bath, and when I get back to him, I do a PR review! I think you're going to turn the whole human race into workaholics.
- Yes, I was too much of a workaholic and my coworkers were worried about me.
integrated1453 I'm afraid that to implement Devin in a company, you need to clarify the scope of utilization and learn the rules and processes... We, too, are currently scoping out and gaining experience with development that accesses core back-end logic and credentials/data.
kos_kim_8 "Improve the skills of the people you ask." Throw some messy stuff out there and see how it fails."
hisayuki_mori If it looks like we are going to exceed 10 ACUs, write the current status in the PR and start the next session and hand over to another Devin session. This looks useful
kgsi I like @teramotodaiki's idea because I work on the side too... It's worth it just to do the technical verification anyway. Dakuon_Findy It's easier to proceed with personal development. bicstone_me I feel that Devin lowers the hurdle for ideas that I want to do and increases the number of attempts, which I feel is a great benefit.
Tomodo_ysys I know, AI development tools are really innovative in that they enable "development while doing housework while raising children".
smasato I would ask Devin for a lunch break or before going to the bathroom.
souhei If you're freelancing and taking work on an hourly basis, you're not getting paid more, even if you pay Devin out of your own pocket and increase your productivity.
- In general, when "work" is measured in hours, there's no incentive to be productive.
souhei Is this what happened to Gyazo earlier? https://note.com/teramotodaiki/n/ndd3b68443702… And I guess this is the story of the 40,000 yen meltdown. https://scrapbox.io/nishio/Devin%E3%81%A74%E4%B8%87%E6%BA%B6%E3%81%8B%E3%81%99%E6%96%B9%E6%B3%95… #devin_findyscrapbox.io How to Melt 40,000 on Devin - Yasukazu Nishio's Outside Brain I thought "What a big deal" at the meeting to see Devin, but I was busy on a business trip and didn't look at it carefully. I don't intend to blame him for that. If we introduce this system in the company, we may have similar cases one by one in each company.
smasato The internal API response that Devin's web app is calling is pretty interesting ......
js4000all "A person who can use it well has tried a lot of things that ask for more than just the right things. You just have to learn it." "Ask Small. Ask big and you'll melt 40,000."
dai___you No matter how big it gets, the stance of the top management (Mr. Wu) to catch up with the needs and utilization methods of users in the field is so wonderful!
nishio Wu is a pretty strong player in competitive programming, although he was not mentioned in the delivery
nishio Oh, this is an interesting topic, but I failed to swing it as a moderator. But if the documentation and knowledge is not well maintained, education is necessary because people will search and read files, just like humans.
kos_kim_8: How effective is this for an existing code base?
js4000all "If I ask them that there should be code like this somewhere, they'll lick the repository and even give me a report." Good
- The actual code reading document generated by Devin is miscellaneously placed here
- https://github.com/nishio/code-reading-docs/blob/main/cline_analysis.md
Can you please stop pushing private keys? - Devin Observation Diary｜Daiki Teramoto
kos_kim_8 Devin will only let you touch private stuff that he handles. public stuff will be forked and made into a private environment.
- Personally, I would like to share information as openly as possible, but I feel that I have no choice but to operate privately at present.
- The flow I wrote in "Eventually, you won't be able to see it." and "[As quality improves, it becomes invisible.
mkt8530 Devin I laughed hysterically when you said you were a freshman in college (not a laughing matter).
ginjih devin's assignment is soooo funny and he acts like a bright college student in IT, which is cute all the way around.
shoudai_magna It's going to feel like asking a young person for a job.
In the conversation, when I pointed out that he had leaked the secret key outside, he erased the code and told me that I should make the key invalid and rebuild it for what I had fixed.
All the contractors I worked with at my last company were like that. I thought they could already be bribed by AI.
I want you to use Gyazo, so can you register first? - Devin Observation Diary｜Daiki Teramoto
- How to Melt 40,000 in Devin
ryomiya1230 You're not smart enough to delegate big tasks on the web.
kos_kim_8 I think it's the same for other AI tools that we should have a sense of the size of the task we're feeding the AI...
- Many people misinterpret it as being too large, so you should consciously try to make it smaller.
kos_kim_8 It's important to use different AI agents; Devin is smart, but does it take a long time per rally to make a skinny advance?
- Devin still has a "ask someone different" feel.
kos_kim_8 human scale out and scale in. It was hard to hire according to the busyness of the project. MH4GF Interesting story about Devin being able to scale out/scale in labor - a colleague who can split infinitely.... .w
katsuhisa__ Devin's work-friendly environment is also work-friendly for all mankind, wise words. imamura_ko_0314 AI agents are well maintained and easy to work with ≒ Newcomers are also easy to work with
katsuhisa__ If you want to follow exactly how Devin behaves in response to instructions, give the instructions in the Devin workspace, not from Slack. I see! I see! js4000all "Devin's workspace has a planner that allows you to visualize in real time the details of what he's trying to do and how he's trying to set it up. It's a great tool!
nishio I failed to mention this, but "I can summon an unlimited number of users with no knowledge to user test my service, and I know everything they are thinking" lowers the bar for user testing I think it lowers the hurdle of user testing very much.
smasato: I had fun trying to get Devin to use our services.
souhei If you ask them to do something messy, they try to solve it in a rustic way. kos_kim_8 "Devin knows, he just doesn't do it" "He does it better when I show him the example."
nishio When I heard the strategy of "only letting people who have a sense of cost use the system," I thought, "I see, but people who have a sense of cost and think the expected value is positive will sign up and use the system themselves. I felt that those who say "I want to try it out, so let's have the company introduce it" may be unqualified at that point because they are not aware of the cost. nishio Well, if you think about it a little more, it's not like that, but if the expected value when combined with "the company organization you belong to" or "the company's source code" is high, there is an incentive to take that action because the expected value of introducing it to the company is positive. If the expected value is high when combined with "your company organization" or "your company's source code", then there is an incentive to take that action because the expected value is positive.
This page is auto-translated from [/nishio/Devin使ってみてどうだった？ ～活用事例と導入時のポイント～](https://scrapbox.io/nishio/Devin使ってみてどうだった？ ～活用事例と導入時のポイント～) using DeepL. If you looks something interesting but the auto-translated English is not good enough to understand it, feel free to let me know at @nishio_en. I'm very happy to spread my thought to non-Japanese readers.