Ahmad - Bit of an emergency, I have a node that...
Bit of an emergency, I have a node that seems to be running on an infinite loop and no way to stop it.
Is there a kill switch for nodes?
It's creating endless rows in a database.
Solution:Jump to solution
Hi @Gaurav Chadha Looks like things got stable. I had to delete everything and recreate it as well as put a few checks in place to prevent the issue.
If it helps below is what I think the problem was:
- I was creating rows from an array and was counting the number of items in the array to repeat the loop...
14 Replies
It will kill/terminate the process after 5 minute automatically, we'll be adding the functionality to force stop instantly.
Thank you for the quick response @Gaurav Chadha . It looks like it's been 10 minutes and still creating new rows.
Is there another way to stop it?
Could you please share it via support form? We'll look now
Or DM the project ID.
@Ahmad We have stopped the execution of the workflow, you can refresh and check it again.
Also, it would be really helpful if you could share how you reached the infinite loop condition?
Thank you!!! Really appreciate the super fast response on this.
I'm not too sure. That same node was working perfectly fine, then when I placed it in the branch something seemed to have broken and the infinite loop occurred.
I suppose it could be the code within the node itself as it does have a loop function, but the fact that it was previously working fine makes this a bit odd.
@Gaurav Chadha looks like I managed to replicate the issue again...it's on an infinite loop once again. This time I at least added a max number of entries but the execution seems to still be happening.
Oh, you may have to delete the workflow to stop it then. Copy the workflow and paste it to a new workflow before deleting. Deleting should stop the deployed version.
Deleted and re-created the flow, but the node still has the little circling icon, does that imply it's still running?
In the new flow? Can you delete the node which is in loop and add it again?
I ended up eliminating the node all together and trying a different method. But just want to make sure ther'e nothing still running in the bakground eating up execution hours.
Okay, once deleted the workflow it should not be running executions any more. We'll ensure to add a a step to force stop execute soon.
...your going to hate me. I seem to keep finding similar issues even though this time I did the following:
- Triggered a repeat loop based on the number of items counted in an array object
- Array had 4 objects
- Counter node somehow counted 754
- Loop triggered and it's now creating 700+ rows
I tried stopping it by first deleting the entire flow and recreating flow #2.
- Then deleted the node, replicated the flow again, this time flow #3 and deleted flow#2
Still running and creating rows
Hi @Ahmad, following up on this, are the rows still getting created? I just now re-checked your workflow "response creator" the node execution in stopped.
Solution
Hi @Gaurav Chadha Looks like things got stable. I had to delete everything and recreate it as well as put a few checks in place to prevent the issue.
If it helps below is what I think the problem was:
- I was creating rows from an array and was counting the number of items in the array to repeat the loop
- The node that was outputting the array was outputting it as a string
- That resulted in the count of number of array items to be miscalculated...700+ instead of 5/6
Once I was able to identify this I put extra measure and check in place like added branches and logic checks to ensure tha the number of items found in the array never exceeded a certain amount.
Luckily the table I'm working with is still in testing, otherwise that could have been bad.
Thanks again for all your help!
Something NASA does for all their code is that any loop which has an undefined iteration length they will add an extra check against an upper limit so it's impossible to infinite loop. This kind of stuff is pretty important when working on cloud infrastructure because one single infinite loop can max out your monthly allowance
Thanks for sharing.