| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-02 | |||
| 16:58:38 | dansmith | ah yeah | |
| 16:58:58 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: WIP: Hey let's support routed networks y'all! https://review.opendev.org/656885 | |
| 17:00:05 | sean-k-mooney | it looks like it should be a 403 here which i dont think is correct either https://review.opendev.org/#/c/674726/3/nova/exception.py | |
| 17:00:21 | sean-k-mooney | ill wait till the jobs completes and check the api log | |
| 17:00:29 | sean-k-mooney | the 500 might be a tempest error code | |
| 17:00:56 | sean-k-mooney | its hard to tell just form the job console | |
| 17:16:48 | Sundar | sean-k-mooney: The error response comes from https://review.opendev.org/#/c/674726/3/nova/api/openstack/compute/servers.py@758 . I see other similar operations grouped in that segment. | |
| 17:17:25 | sean-k-mooney | the job just finished https://zuul.opendev.org/t/openstack/build/0f144bf83a614d31938d353a1bb05935 | |
| 17:17:37 | sean-k-mooney | im going to confirm if the 500 is from tempest or nova | |
| 17:17:45 | Sundar | I have tested it from the 'openstack server ...' CLI, so I don;t think it is tempest | |
| 17:18:26 | sean-k-mooney | well the "computefault" bit makes me think its from tempest | |
| 17:19:12 | Sundar | You would see a 500 error if you do 'openstack server suspend foo'. | |
| 17:19:34 | sean-k-mooney | its not a server error so it should not be a 500 | |
| 17:19:46 | sean-k-mooney | its a client error because you requested an unsupported operation | |
| 17:20:16 | sean-k-mooney | so it s houl be either a 400 "bad request" 409 "conflict" or 405 "method not supported" | |
| 17:20:31 | sean-k-mooney | proably a 400 or 409 we dont use 405 as far as i know | |
| 17:23:28 | sean-k-mooney | Sundar: so yes we are raising the exeption but it is not being caught and its being converted into a 500 http://paste.openstack.org/show/790209/ | |
| 17:25:55 | openstackgerrit | Merged openstack/nova master: Define Cyborg ARQ binding notification event. https://review.opendev.org/692707 | |
| 17:26:23 | Sundar | sean-k-mooney: Hmm, let me see what's happening with https://review.opendev.org/#/c/674726/3/nova/api/openstack/compute/servers.py@758 | |
| 17:27:12 | sean-k-mooney | the block operation patch needs to be rewored to only block operation that dont work | |
| 17:28:38 | Sundar | sean-k-mooney: Unless I have proven or validated that an op works, I have blocked it. There are many ops that I have not validated, such as backup. Plus, there are grounds to suspect such ops won't always work, since there is state in the accelerator. | |
| 17:28:51 | sean-k-mooney | Sundar: yep which im hard -1 on | |
| 17:29:13 | sean-k-mooney | im ok with blocking ops we know wont work | |
| 17:29:28 | sean-k-mooney | im not ok with blocking ops just because we feel like it. | |
| 17:29:48 | Sundar | It is safer to block an op than let it loose and document that it is untested with accelerators. I thought we had discussed that. | |
| 17:30:23 | Sundar | Not 'we feel like it;, but whether it is tested and whether there are grounds to believe it will work dafely all the time | |
| 17:30:26 | sean-k-mooney | we agreed not to block the ops and just document them | |
| 17:30:28 | Sundar | *safely | |
| 17:30:41 | sean-k-mooney | that is why https://review.opendev.org/#/c/674726 was abandoned | |
| 17:30:55 | Sundar | Hmm, I recall otherwise. Let me look for past IRC discussions. | |
| 17:31:39 | sean-k-mooney | we can bring it back but i dont think its ok to block things like rebuild or volume based snapthot when we know they should work and they do work at least with the fake driver | |
| 17:39:22 | Sundar | sean-k-mooney: Here is our past discussion: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-02-07.log.html#t2020-02-07T19:17:13 . I see different opinions, but you still want the doc-only approach. But you do agree that people won't read the docs till things break | |
| 17:39:23 | Sundar | (http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-02-07.log.html#t2020-02-07T19:31:39) | |
| 17:39:54 | Sundar | Is it not safer to block an unproven op (but only for accelerators obviously)? | |
| 17:40:09 | Sundar | I am not si sure that snapshots will work | |
| 17:40:11 | sean-k-mooney | that is form last month not form when we deiced not to block everything before christmas | |
| 17:40:24 | sean-k-mooney | a live snapshot will work | |
| 17:40:41 | sean-k-mooney | a cold snapshot might should too | |
| 17:40:53 | sean-k-mooney | since it stops the vm and then starts it after | |
| 17:41:37 | Sundar | A snapshot is going to save the mmap'd buffers in VM memory, and bring it back on restore. But the accelerator state would have changed in between as the VM continued op. So, on restore, the buffers in the VM will not match the accelerator contents. | |
| 17:42:07 | sean-k-mooney | Sundar: no it wont | |
| 17:42:21 | sean-k-mooney | a snappshot only makes a copy of the root disk | |
| 17:42:38 | sean-k-mooney | its not the same as a vm snapshot in virtual box or hyperv | |
| 17:42:44 | Sundar | I thought there's an option to do snapshot with RAM. | |
| 17:42:51 | sean-k-mooney | no | |
| 17:42:57 | sean-k-mooney | not in the libvirt dirver | |
| 17:43:10 | sean-k-mooney | you can do that with qemu but not via nova | |
| 17:43:13 | Sundar | libvirt allows it -- but apparently not the Nova libvirt driver? | |
| 17:43:27 | Sundar | I see | |
| 17:43:38 | sean-k-mooney | yes becasue we have no where to store the ram dump | |
| 17:44:31 | sean-k-mooney | we could in theory upload it as a second image to glace or something but snapshot in the nova api is just of the disk | |
| 17:46:00 | Sundar | Are you able to check snapshot with FPGA-attached VMs? I'll try too -- but what works in the lab may not always work in the field. | |
| 17:46:41 | sean-k-mooney | i have not deployed on real hardware so no. | |
| 17:48:40 | Sundar | I would say that it is safer to block something rather than speculatively allow it. Do you have users who strongly need snapshots with accelerators? | |
| 17:49:43 | sean-k-mooney | not specifically but i would not consider movingit form techpreview without it | |
| 17:50:10 | sean-k-mooney | in the curent patch there are no ways to do any move operation and you cant resuceu or snapshot the vm | |
| 17:50:28 | sean-k-mooney | so there is 0 way to do maintaince on the host or recover in the case of a vailure | |
| 17:50:33 | sean-k-mooney | even evacuate is blocked | |
| 17:50:58 | sean-k-mooney | as is adding or removeing volumes or interfaces | |
| 17:51:22 | sean-k-mooney | so with only the abilty to boot and delete a vm we would not support this in productin for customer to use | |
| 17:52:21 | sean-k-mooney | thats a downstrema desciion however. it wont affect if we support this in upstream | |
| 17:52:56 | Sundar | I see your concern. The real solution is to test these ops well with real accelerators and real-ish workloads (not sample bitstreams). I'll pass that on to my Cyborg/Intel colleagues. | |
| 17:53:22 | Sundar | Meanwhile, based on your input, I could remove the blocks on snapshot and rescue. Presumably backup too? | |
| 17:54:16 | sean-k-mooney | backup is basicaly a cron job in the nova code that calls snapshot and clean up old snapshots | |
| 17:54:27 | sean-k-mooney | so if snapshot work then back will | |
| 17:55:24 | sean-k-mooney | im not sure about rescue. it should work but i dont know if the acclerator will be aviable while its in resuce state | |
| 17:55:53 | sean-k-mooney | if the accelerator in not present in the vm during or after resuce has been completed then we might need to block it | |
| 17:56:32 | Sundar | Since the rescue brings in a different disk image, which may not have the right device drivers, it probably will not. The unrescue to the previous image "should" work. | |
| 17:57:04 | sean-k-mooney | oh the other hand not everyone would need the acclerator to be present during resuce so as long as we dont loose itafter unrescue we should be ok | |
| 17:57:23 | Sundar | Ok. So rescue is in. What about suspend? It doesn't work today because we don't have the code to handle accelerators in that path. | |
| 17:57:51 | sean-k-mooney | suspend wont work without a code change so ya that can remain blocked | |
| 17:58:31 | openstackgerrit | Mark Goddard proposed openstack/nova master: Fix PCI passthrough race on reschedule (claims) https://review.opendev.org/710847 | |
| 17:58:32 | openstackgerrit | Mark Goddard proposed openstack/nova master: Fix PCI passthrough race on reschedule (refresh) https://review.opendev.org/710848 | |
| 17:59:23 | Sundar | sean-k-mooney: The other 2 ops mentioned in https://review.opendev.org/#/c/674726/3/nova/tests/functional/test_servers.py are rebuild and resize. | |
| 17:59:42 | dansmith | Sundar: sean-k-mooney just catching up, but if we can't support snapshot, that's a huge problem IMHO (although I like sean-k-mooney think it's likely not a problem) | |
| 18:00:19 | dansmith | if rescue doesn't include the accelerator I don't think we should block it | |
| 18:00:33 | dansmith | rescue is a special environment | |
| 18:00:46 | dansmith | I don't really know why it wouldn't include it, but I don't think that's a major problem or anything | |
| 18:01:02 | sean-k-mooney | ya i think rescue should be fine just as long as we validate the acclertor is present after unrescue when we go back to the stanard env | |
| 18:01:21 | dansmith | yes, that's obviously an important point :D | |
| 18:01:34 | sean-k-mooney | Sundar: resize wont work | |
| 18:01:49 | sean-k-mooney | i tested that with the fake dirver and it does nto update placmenet correctly | |
| 18:01:57 | sean-k-mooney | rebuild should work fine | |
| 18:04:01 | sean-k-mooney | Sundar: dansmith i left comment on https://review.opendev.org/#/c/674726 with what i think should and should not work | |
| 18:10:23 | sean-k-mooney | Sundar: by the way im going to be AFK or on PTO for most of the rest of this week so if i dont repond or am offline that is why | |
| 18:17:38 | Sundar | sean-k-mooney: dansmith: I'll respond in the patch. Meanwhile I have filed a Cyborg storyboard task: https://storyboard.openstack.org/#!/story/2007358 . I'll raise that with Cyborg folks. | |
| 18:18:34 | sean-k-mooney | Sundar: is there a thirdpart hardwar ci somewhere? | |
| 18:18:54 | sean-k-mooney | if so they could just do what im doing in the multinode job | |
| 18:19:15 | sean-k-mooney | that will validate all the lifecycle ops | |
| 18:19:55 | sean-k-mooney | they could blacklist the tests they know do not work and as the missing ops get adressed they can be removed form the blacklist | |
| 18:20:19 | Sundar | sean-k-mooney: Unfortunately, our current plans for 3pCI with FPGAs are nixed. Our colleagues in other parts of Intel could possibly resume it, since we have the hardware lined up. | |
| 18:20:24 | sean-k-mooney | alternitvly the cyborg plugin could be extended but that really up to whowever is going to test it | |
| 18:20:47 | sean-k-mooney | ok ya i kind of guessed that would be the case | |
| 18:22:25 | Sundar | sean-k-mooney: Noted your PTO. Thanks for your help. I'm not sure how long I'll be around after this week. | |
| 18:34:29 | sean-k-mooney | ok time for dinner. | |
| 18:34:43 | sean-k-mooney | o/ | |
| 19:33:25 | openstackgerrit | Merged openstack/nova master: Fix os-attach-interfaces policy to be admin_or_owner https://review.opendev.org/705135 | |
| 19:50:53 | openstackgerrit | David Hill proposed openstack/nova stable/queens: Port binding based on events during live migration https://review.opendev.org/710861 | |
| 19:51:09 | openstackgerrit | David Hill proposed openstack/nova stable/queens: Port binding based on events during live migration https://review.opendev.org/710861 | |
| 20:21:51 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Use virDomainBlockCopy to swap volumes with >= 5.10.0 https://review.opendev.org/696834 | |