Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-04-20
10:57:43 gibi bauzas: I have started looking at https://review.opendev.org/#/c/712741/ just now
11:00:57 brinzhang gibi: I think your think make sense, agree, thanks
11:03:14 brinzhang this is an invalid bug
11:22:13 songwenping gb:Hi gibi. I am working on nova-cyborg-interaction now, and commit this patch https://review.opendev.org/#/c/720670/
11:22:38 gibi songwenping: hi
11:22:50 songwenping We haven't show the ARQ id in dashboard now.
11:23:09 songwenping But i think we will show it like cinder volume.
11:24:05 songwenping So should we handle the cyborg exception after showing it?
11:24:06 gibi songwenping: still, expecting the end user to _know_ where and how to clean up after a seemingly _successfull_ server delete operations feels bad
11:24:58 gibi basically after every server delete the end user would need to check the cyborg API to know if the ARQs are freed up or not
11:25:09 gibi I don't like that
11:26:20 brinzhang gibi, songwenping: agree with gibi, if there are so many resources leaked in Cyborg, it will be a heavy works to cleanup
11:26:49 brinzhang but compared with Cinder logical, it also has the same issue.
11:27:32 brinzhang maybe we shuold have a logical to deal with this, or dealed in Cinder and/or Cyborg
11:33:23 songwenping gb:Yeah, it's indeed a problem leaking many resources in system. I just want to keep pace with cinder logical.
11:36:57 gibi songwenping: what is the use case you want to solve? you mentioned deploy and undeploy cyborg. there I think before undeploy the admin needs to clean up the cyborg users. Also mentioned failure in cyborg. If that failure is intermittent (e.g service restart or network interrupt) then I think end user needs to retry the delete. if the cyborg failure is static then that is a cyborg but to be fixed
11:37:20 gibi s/but/bug/
11:43:24 songwenping_ gibi:i want to solve the second use case.
11:45:33 brinzhang gibi: I give you use case from my customer
11:45:45 brinzhang s/use case/ a use case
11:45:54 brinzhang gibi: Due to the system upgrade, the cyborg service cannot be started. If the user wants to clear the instance that contains ‘accel: device_profile_name’ in the flavor, the instance will be in an error state and cannot release scarce resources such as GPU and FPGA. If that is the user's only resource, it may also be considered for manual cleaning. This is common for small customers.
11:47:25 brinzhang Of course, this is a scene of its edge.
11:47:32 gibi brinzhang: so during and upgrade some of the control plane services are still up (e.g. nova) but some of them are down (e.g cyborg)
11:48:26 brinzhang This may be a treatment, but it is not so perfect.
11:48:33 gibi but if cyborg is down then who the user could ever free up FPGA resources?
11:48:43 gibi s/who/how/
11:49:31 gibi also even if it is freed up it cannot be used again as cyborg is down
11:49:51 brinzhang maybe that canbe done in db, this is perhaps the worst case
11:51:07 gibi sorry but I my mind if cyborg service is down, then the user cannot and should not do anything with resources managed by cyborg
11:51:13 brinzhang gibi: I just put forward such a scenario, and I agree with you, your consider is right
11:52:05 gibi then we agree that this use case is not valid :)
11:52:12 songwenping_ agree with gibi.
11:53:27 brinzhang yes, but I think I also need to think how to deal with this scenario, we did encounter this situation.
11:55:22 brinzhang another way, that can power off the server, and migrate its instance, than re-deployed the OpenStack
11:55:33 brinzhang in a new region
11:57:55 gibi honestly I don't see why does your deployment need to support manipulating FPGAs while cyborg service is doewn
11:57:58 gibi down
12:01:27 brinzhang yes, it isnot make sense. good bye gibi, hope you have a good day ^
12:05:15 gibi brinzhang: have a nice afternoon
12:12:03 bauzas gibi: ack thanks
12:12:21 bauzas brinzhang: if you want, ping me tomorrow for discussing about something about reviews
12:12:36 bauzas brinzhang: (UTC+2 here)
12:40:16 gibi bauzas: will there be a FUP for the comments from lyarwood in https://review.opendev.org/#/c/712118/ ?
12:40:47 bauzas gibi: haven't seen them yet
12:41:15 bauzas gibi: FWIW, in https://review.opendev.org/#/c/712118/ like I said in a comment, this change is no longer needed for https://review.opendev.org/#/c/712741/
12:42:14 gibi bauzas: will you then remove it from the series?
12:42:21 bauzas gibi: for the comment nits, sure I can do it in a FUP
12:42:41 bauzas gibi: we can merge it since I already provided a ML thread for out-of-tree drivers maintainers
12:42:41 gibi bauzas: yes, comment nits are totally FUPable but if the whole patch is not needed then it is even better
12:43:11 bauzas gibi: or wait, I'll rebase this one on top of https://review.opendev.org/#/c/712741/ and just provide a new revision for the nits
12:43:22 bauzas will be done in 1 min
12:43:29 gibi but then the allocation would be an unused param
12:43:31 gibi isn't it?
12:46:19 bauzas gibi: for finish_revert_migration() yes
12:46:31 bauzas gibi: to clarify, I'll just provide the new series
12:46:58 bauzas and people can discuss on the opportunity to merge https://review.opendev.org/#/c/712118/ if nothing uses the new param or not
12:47:14 gibi bauzas: OK, I will check the new series
12:47:57 bauzas gibi: should be done in 5 mins, just verifying unittests and functests because of a minor merge conflict
12:49:18 gibi cool
13:00:02 francoisp gibi hello, we would need an external reviewer (outside of RH) to check on https://review.opendev.org/#/c/669674/ , would you have time to have a look?
13:00:12 bauzas gibi: excellent concern FWIW https://review.opendev.org/#/c/712741/6/nova/tests/functional/libvirt/test_vgpu.py@45
13:03:32 gibi francoisp: based on a recent agreement you only need to keep the two company rule for high impact changes, anything that
13:03:35 gibi involves a microversion, service version, rpc version, or database
13:03:36 gibi migration.
13:03:45 gibi francoisp: but sure I will look at that bugfix
13:04:50 gibi bauzas: honestly I failed to prove that it can actaully cause any problem but if you can add someting to the setUp to reset the test object level variable that could scratch my itch
13:04:52 openstackgerrit Sylvain Bauza proposed openstack/nova master: Allocate mdevs when resizing or reverting resize https://review.opendev.org/712741
13:04:53 openstackgerrit Sylvain Bauza proposed openstack/nova master: Pass allocations to virt drivers when reverting resize https://review.opendev.org/712118
13:05:12 francoisp thanks very much gibi
13:05:36 gibi francoisp: for reference http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013553.html
13:07:29 lyarwood gibi: I asked for additional review outside of RH in francoisp's case as that change impacts all callers to cinder across all virt drivers.
13:07:42 francoisp ok thanks gibi, that makes sense, otherwise you would get overwhelmed
13:07:53 gibi lyarwood: I see that is reasonable
13:22:38 bauzas gibi: if you don't mind reapplying your +2 on the vgpu resize change given the only change was due to a merge conflict resolution https://review.opendev.org/#/c/712741/6..7
13:27:35 gibi bauzas: done. and thanks for writing the prelude
13:27:45 bauzas ta
13:31:26 gibi francoisp, lyarwood: +A-d the cinder retry
13:32:25 francoisp ok great, thank you gibi
13:32:48 lyarwood yup thanks gibi
13:37:43 openstackgerrit jayaditya gupta proposed openstack/nova master: Support for --force flag for nova-manage placement heal_allocations command https://review.opendev.org/715395
13:47:29 nightmare_unreal mriedem: thanks for the review :) As you have suggested I have made changes accordingly but I am still facing 1 issue. It seems the allocated ram ( bogus ram) won't change if you call heal allocation with force flag or without force flag :/ I have added comments for it https://review.opendev.org/#/c/715395/10
13:47:30 nightmare_unreal Thanks
14:05:25 gmann melwitt: stephenfin gibi seems like johnthetubaguy is not online. how we should proceed on these last bits to merge as 23rd is hard string freeze - https://review.opendev.org/#/q/topic:bp/policy-defaults-refresh+status:open
14:06:29 gibi gmann: I'm on a call I will ping back in an hour
14:06:48 gibi but overall I can try to spend 1 hour on those today
14:08:41 gmann gibi: thanks
14:26:55 openstackgerrit sean mooney proposed openstack/nova-specs master: move implemented spec for train https://review.opendev.org/706276
14:27:50 openstackgerrit sean mooney proposed openstack/nova-specs master: move implemented spec for train https://review.opendev.org/706276
14:29:53 mriedem nightmare_unreal: that's the point of the feature, correct? if it's not working you're going to need to debug it.
14:30:05 mriedem but that's why i asked for that kind of test
14:31:42 nightmare_unreal yeaah
14:35:23 openstackgerrit sean mooney proposed openstack/nova-specs master: move implemented spec for ussuri https://review.opendev.org/721278
14:40:53 kashyap lyarwood: Hey, do you have the reproducer for that 'q35' thing on Ubuntu?
14:41:26 kashyap Ah, you updated this morning
14:42:43 kashyap Okay, it looks like a nudge. (Because, I don't see any 'diff' b/n 3..4: https://review.opendev.org/#/c/708701/3..4/.zuul.yaml)
15:02:15 lyarwood kashyap: sorry was hacking away on something downstream
15:02:25 lyarwood kashyap: I've just rebased that today to see if it still reproduces
15:02:36 lyarwood kashyap: I don't have anything written up, I only manually reproduced it before.
15:02:37 kashyap lyarwood: No problem; I asked on the change
15:02:44 kashyap (I don't count on instant responses :))

Earlier   Later