Running step: extraction 50%|██████████████████████████████████████████████████████ | 2/4 [00:05<00:05, 2.92s/it]JSON error: Expecting value: line 1 column 1 (char 0) Input was: 外国勢力の進行に対する厳罰 Response was: I'm sorry, but I can only assist in English. If you provide the text in English, I'll be happy to help you refine it. Retrying... 75%|█████████████████████████████████████████████████████████████████████████████████ | 3/4 [00:07<00:02, 2.51s/it]JSON error: Expecting value: line 1 column 1 (char 0) Input was: 統治機構改革 Response was: I'm sorry, but it seems like the text "統治機構改革" is in a language that I am not able to understand. Could you please provide the text in English so that I can assist you with making it more concise and easy to read? Retrying... JSON error: Expecting value: line 1 column 1 (char 0) Input was: 統治機構改革 Response was: I'm sorry, but it seems like the text "統治機構改革" is in a language other than English, and I am unable to provide a concise and clear argument based on it. If you could provide a translation or more context, I would be happy to assist further. Retrying... JSON error: Expecting value: line 1 column 1 (char 0) Input was: 統治機構改革 Response was: I'm sorry, but it seems like the text you provided is not related to the topic of artificial intelligence. Could you please provide an argument related to AI for me to help you refine? Retrying... JSON error: Expecting value: line 1 column 1 (char 0) Input was: 統治機構改革 Response was: I'm sorry, but it seems like the text "統治機構改革" is in a language other than English. Could you please provide a translation or more context so I can assist you better? Silently giving up on trying to generate valid list. 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:12<00:00, 3.23s/it]

Running step: embedding 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:10<00:00, 10.26s/it]

Running step: clustering Traceback (most recent call last): File "/Users/nishio/talk-to-the-city-reports/scatter/venv/lib/python3.12/site-packages/nltk/corpus/util.py", line 84, in __load root = nltk.data.find(f"{self.subdir}/{zip_name}") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/nishio/talk-to-the-city-reports/scatter/venv/lib/python3.12/site-packages/nltk/data.py", line 583, in find raise LookupError(resource_not_found) LookupError: ********************************************************************** Resource stopwords not found. Please use the NLTK Downloader to obtain the resource: >>> import nltk >>> nltk.download('stopwords')

stop = stopwords.words("english") vectorizer_model = CountVectorizer(stop_words=stop)

Running step: clustering OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead. 2024-05-30 14:43:23,417 - BERTopic - Reduced dimensionality 2024-05-30 14:43:23,423 - BERTopic - Clustered reduced embeddings

Running step: labelling 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:02<00:00, 1.20it/s] Running step: takeaways 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:04<00:00, 1.67s/it] Running step: overview

Running step: translation Translating to Mandarin... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:16<00:00, 2.38s/it] Translating to English... 0%| | 0/7 [00:00<?, ?it/s]JSON error: Expecting ',' delimiter: line 1 column 216 (char 215) Response was: ["Through public debate, considering Japan's history and education, it is necessary to create a democratic law by taking time into account.", "It is important to reflect the public opinion by creating a process of "constitutional amendment" through consensus.", "It is necessary to stop the annual budget system.", "For the protection of human rights, it is essential to have discussions and collaborations on the internet.", "Through public debate, considering Japan's history and education, it is necessary to create a democratic law by taking time into account.", "By passing the process of "constitutional amendment", reflecting the public opinion is important.", "For Japan to consider Japan's future, it is necessary to establish a law that will become the basis.", "By revising the 41st article of the constitution, implementing the national referendum, and reflecting its contents in politics is important.", "Keep an eye on the "constitutional amendment".", "Revise the 41st law, implement the national referendum, and reflect its contents in politics."] Retrying batch... Warning: batch size mismatch! Batch len: 10 Response len: 9 Batch item 0: 国民の議論を通じて、日本の伝統や習慣を考慮した自主憲法を時間をかけて作るべき Response: By engaging in public debate, considering Japan's history and education, creating necessary self-defense laws over time is essential Batch item 1: 「創憲」のプロセスを通じて国民の意見を反映させるべき Response: By promoting the principles of 'innovation', reflecting the opinions of the people to be considered Batch item 2: 廃止すべき予算の単年度制度 Response: It is necessary to stop the annual budget system Batch item 3: 人権保護のためにインターネット上での取り組みが必要 Response: For the protection of human rights, it is essential to have an internet platform for gathering information Batch item 4: 国民の議論を通じて日本の伝統や習慣を考慮した自主憲法を時間をかけて作るべき Response: By engaging in public debate, considering Japan's history and education, creating necessary self-defense laws over time is essential Batch item 5: 『創憲』のプロセスを通じて国民の意見を反映させるべき Response: Establishing laws to become a better future for Japan that the Japanese people consider Batch item 6: 日本国民が考える日本の将来のためになる憲法を創設すべきだ。 Response: By revisiting discussions and debates, developing Japan's unique education and history, creating necessary self-defense laws over time Batch item 7: 国民と議論を重ねながら、日本独自の習慣や伝統を鑑みた自主憲法を時間をかけて作る。 Response: Keep the focus on 'innovation' Batch item 8: 「創憲」を目指す。 Response: Amend the 41st law, implement national referendums on important legislation, and reflect its contents in politics Batch item 9: 憲法第四十一条を改正して、重要法案について国民投票を実施し、その内容を政治に反映させるべき。 Retrying with smaller batches...

Running step: aggregation Running step: visualization > next-app@0.1.0 build > next build

{ "name": "Recursive Public, Agenda Setting", "question": "What should be top of the agenda as humanity develops and deploys artificial intelligence?", "input": "UTAS_2022__SQ6_16_FA", "model": "gpt-3.5-turbo", "extraction": { "workers": 3, "limit": 12 }, "clustering": { "clusters": 3 }, "translation": { "model": "gpt-4", "languages": ["Mandarin", "English"], "flags": ["TW", "EN"] }, "intro": "This AI-generated report relies on data from the UTAS 2022 (東京大学谷口研究室・朝日新聞社共同調査)." }

So, here is what I am planning to run: {'step': 'extraction', 'run': False, 'reason': 'nothing changed'} {'step': 'embedding', 'run': False, 'reason': 'nothing changed'} {'step': 'clustering', 'run': False, 'reason': 'nothing changed'} {'step': 'labelling', 'run': False, 'reason': 'nothing changed'} {'step': 'takeaways', 'run': False, 'reason': 'nothing changed'} {'step': 'overview', 'run': False, 'reason': 'nothing changed'} {'step': 'translation', 'run': True, 'reason': 'some parameters changed: languages'} {'step': 'aggregation', 'run': True, 'reason': 'some dependent steps will re-run: translation'} {'step': 'visualization', 'run': True, 'reason': 'some dependent steps will re-run: aggregation'} Looks good? Press enter to continue or Ctrl+C to abort.

File "/Users/nishio/talk-to-the-city-reports/scatter/pipeline/steps/translation.py", line 102, in translate_batch parsed = [a.strip() for a in json.loads(response)]

comments = pd.read_csv(f"inputs/{config['input']}.csv") limit = config['extraction']['limit'] comment_ids = (comments['comment-id'].values)[:limit]

99%|███████████████████████████████████████████████████████████████████████████████████████████████████████▏| 666/671 [29:36<00:13, 2.67s/it] Traceback (most recent call last):

100%|██████████████████████████████████████| 666/666 [30:12<00:00, 2.72s/it] Running step: embedding 100%|██████████████████████████████████████| 6/6 [00:15<00:00, 2.65s/it] Running step: clustering OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead. 2024-05-31 04:37:45,187 - BERTopic - Reduced dimensionality 2024-05-31 04:37:45,284 - BERTopic - Clustered reduced embeddings Running step: labelling 100%|██████████████████████████████████████| 7/7 [00:04<00:00, 1.74it/s] Running step: takeaways 100%|██████████████████████████████████████| 7/7 [00:16<00:00, 2.35s/it] Running step: overview

93%|██████████████████████████████████████ | 556/601 [1:33:50<07:35, 10.13s/it] Traceback (most recent call last): File "/Users/nishio/talk-to-the-city-reports/scatter/pipeline/steps/translation.py", line 102, in translate_batch parsed = [a.strip() for a in json.loads(response)] ^^^^^^^^^^^^^^^^^^^^ ... json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 382 (char 381)

% python main.py configs/aipubcom.json (!) {'step': 'clustering', 'filename': 'clusters.csv', 'dependencies': {'params': ['clusters'], 'steps': ['embedding']}, 'options': {'clusters': 8}} step parameter 'clusters' changed from '7' to '50' diff_params ['clusters'] So, here is what I am planning to run: {'step': 'extraction', 'run': False, 'reason': 'nothing changed'} {'step': 'embedding', 'run': False, 'reason': 'nothing changed'} {'step': 'clustering', 'run': True, 'reason': 'some parameters changed: clusters'} {'step': 'labelling', 'run': True, 'reason': 'some dependent steps will re-run: clustering'} {'step': 'takeaways', 'run': True, 'reason': 'some dependent steps will re-run: clustering'} {'step': 'overview', 'run': True, 'reason': 'some dependent steps will re-run: labelling, takeaways'} {'step': 'translation', 'run': True, 'reason': 'some dependent steps will re-run: labelling, takeaways, overview'} {'step': 'aggregation', 'run': True, 'reason': 'some dependent steps will re-run: clustering, labelling, takeaways, overview, translation'} {'step': 'visualization', 'run': True, 'reason': 'some dependent steps will re-run: aggregation'} Looks good? Press enter to continue or Ctrl+C to abort.

100%|████████████████████████████████████████████| 606/606 [2:31:52<00:00, 15.04s/it] Translating to Taiwan... 9%|█████████ | 53/606 [08:00<1:13:18, 7.95s/it]

100%|███████████████████████████████████████████████████▊| 605/606 [1:38:47<00:09, 9.80s/it] Traceback (most recent call last): File "/Users/nishio/talk-to-the-city-reports/scatter/pipeline/steps/translation.py", line 119, in translate_batch return translate_batch(batch, lang_prompt, model, retries - 1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ... RecursionError: maximum recursion depth exceeded python main.py configs/aipubcom.json 56.16s user 33.46s system 0% cpu 4:13:37.44 total

for (i, option) in enumerate(sysargv): if option == '-f': config['force'] = True if option == '-o': config['only'] = sysargv[i + 1] if option == '-skip-interaction': config['skip-interaction'] = True

import pandas as pd # Load the arguments from the CSV file arguments = pd.read_csv("outputs/plurality/args.csv") # List of arguments to ignore ignores = ["error", "argument", "reference", "references", "source", "sources", "message", "contributions"] # show size print(len(arguments)) # Filter out the ignored arguments filtered_arguments = arguments[~arguments['argument'].isin(ignores)] # show size print(len(filtered_arguments)) # write filtered_arguments to output/plurality/args.csv filtered_arguments.to_csv("outputs/plurality/args.csv", index=False)