| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-02 | |||
| 15:26:08 | dansmith | yeah, okay, just thinking about maybe some lighter way to do that | |
| 15:26:13 | sean-k-mooney | dansmith: this is more or less the local.confs im using http://paste.openstack.org/show/789773/ | |
| 15:26:36 | sean-k-mooney | dansmith: ya i have 32G of ram on my laptop so 2 8G vms is fine | |
| 15:26:45 | sean-k-mooney | the config trick might work | |
| 15:26:53 | sean-k-mooney | but it just felt like more effort | |
| 15:26:55 | dansmith | yeah, ram isn't the problem | |
| 15:26:57 | dansmith | could be | |
| 15:29:33 | sean-k-mooney | do you want me to kick off a ci run with the latest set of patches sundar pushed | |
| 15:30:35 | sean-k-mooney | i basically just need to recheck https://review.opendev.org/#/c/709641/ or update the depends-on url to point to the latest patch in the cain | |
| 15:31:11 | dansmith | sean-k-mooney: yeah, would be good I think | |
| 15:31:14 | sean-k-mooney | dansmith: removing the singelton seemed to resolve all the auth issues | |
| 15:31:17 | dansmith | I'm working through those now | |
| 15:31:21 | dansmith | sean-k-mooney: nice :) | |
| 15:32:14 | sean-k-mooney | ok ill point it to pull in the last funcitonal patch in the cain which is the one that block unsupported opperation and we can get the job to assert they all fail/are rejected as we expect | |
| 15:35:32 | sean-k-mooney | cool thats running we will know in about an hour | |
| 15:35:44 | dansmith | schweet | |
| 15:40:09 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-evacuate https://review.opendev.org/708235 | |
| 16:17:11 | openstackgerrit | John Garbutt proposed openstack/nova master: Add a tests to check when legacy access is removed https://review.opendev.org/710813 | |
| 16:25:06 | openstackgerrit | John Garbutt proposed openstack/nova master: Add a tests to check when legacy access is removed https://review.opendev.org/710813 | |
| 16:29:50 | openstackgerrit | Merged openstack/nova master: Add test coverage of existing os-agents policies https://review.opendev.org/701644 | |
| 16:58:09 | sean-k-mooney | dansmith: the test run is still executing but i am seeing some failes as expected. however the error code is 500 which seams incorrect | |
| 16:58:16 | sean-k-mooney | Body: b'{"computeFault": {"code": 500, "message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\\n | |
| 16:58:18 | sean-k-mooney | \'nova.exception.ForbiddenWithAccelerators\'>"}}' | |
| 16:58:33 | sean-k-mooney | it should be a 4XX error correct | |
| 16:58:38 | dansmith | ah yeah | |
| 16:58:58 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: WIP: Hey let's support routed networks y'all! https://review.opendev.org/656885 | |
| 17:00:05 | sean-k-mooney | it looks like it should be a 403 here which i dont think is correct either https://review.opendev.org/#/c/674726/3/nova/exception.py | |
| 17:00:21 | sean-k-mooney | ill wait till the jobs completes and check the api log | |
| 17:00:29 | sean-k-mooney | the 500 might be a tempest error code | |
| 17:00:56 | sean-k-mooney | its hard to tell just form the job console | |
| 17:16:48 | Sundar | sean-k-mooney: The error response comes from https://review.opendev.org/#/c/674726/3/nova/api/openstack/compute/servers.py@758 . I see other similar operations grouped in that segment. | |
| 17:17:25 | sean-k-mooney | the job just finished https://zuul.opendev.org/t/openstack/build/0f144bf83a614d31938d353a1bb05935 | |
| 17:17:37 | sean-k-mooney | im going to confirm if the 500 is from tempest or nova | |
| 17:17:45 | Sundar | I have tested it from the 'openstack server ...' CLI, so I don;t think it is tempest | |
| 17:18:26 | sean-k-mooney | well the "computefault" bit makes me think its from tempest | |
| 17:19:12 | Sundar | You would see a 500 error if you do 'openstack server suspend foo'. | |
| 17:19:34 | sean-k-mooney | its not a server error so it should not be a 500 | |
| 17:19:46 | sean-k-mooney | its a client error because you requested an unsupported operation | |
| 17:20:16 | sean-k-mooney | so it s houl be either a 400 "bad request" 409 "conflict" or 405 "method not supported" | |
| 17:20:31 | sean-k-mooney | proably a 400 or 409 we dont use 405 as far as i know | |
| 17:23:28 | sean-k-mooney | Sundar: so yes we are raising the exeption but it is not being caught and its being converted into a 500 http://paste.openstack.org/show/790209/ | |
| 17:25:55 | openstackgerrit | Merged openstack/nova master: Define Cyborg ARQ binding notification event. https://review.opendev.org/692707 | |
| 17:26:23 | Sundar | sean-k-mooney: Hmm, let me see what's happening with https://review.opendev.org/#/c/674726/3/nova/api/openstack/compute/servers.py@758 | |
| 17:27:12 | sean-k-mooney | the block operation patch needs to be rewored to only block operation that dont work | |
| 17:28:38 | Sundar | sean-k-mooney: Unless I have proven or validated that an op works, I have blocked it. There are many ops that I have not validated, such as backup. Plus, there are grounds to suspect such ops won't always work, since there is state in the accelerator. | |
| 17:28:51 | sean-k-mooney | Sundar: yep which im hard -1 on | |
| 17:29:13 | sean-k-mooney | im ok with blocking ops we know wont work | |
| 17:29:28 | sean-k-mooney | im not ok with blocking ops just because we feel like it. | |
| 17:29:48 | Sundar | It is safer to block an op than let it loose and document that it is untested with accelerators. I thought we had discussed that. | |
| 17:30:23 | Sundar | Not 'we feel like it;, but whether it is tested and whether there are grounds to believe it will work dafely all the time | |
| 17:30:26 | sean-k-mooney | we agreed not to block the ops and just document them | |
| 17:30:28 | Sundar | *safely | |
| 17:30:41 | sean-k-mooney | that is why https://review.opendev.org/#/c/674726 was abandoned | |
| 17:30:55 | Sundar | Hmm, I recall otherwise. Let me look for past IRC discussions. | |
| 17:31:39 | sean-k-mooney | we can bring it back but i dont think its ok to block things like rebuild or volume based snapthot when we know they should work and they do work at least with the fake driver | |
| 17:39:22 | Sundar | sean-k-mooney: Here is our past discussion: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-02-07.log.html#t2020-02-07T19:17:13 . I see different opinions, but you still want the doc-only approach. But you do agree that people won't read the docs till things break | |
| 17:39:23 | Sundar | (http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-02-07.log.html#t2020-02-07T19:31:39) | |
| 17:39:54 | Sundar | Is it not safer to block an unproven op (but only for accelerators obviously)? | |
| 17:40:09 | Sundar | I am not si sure that snapshots will work | |
| 17:40:11 | sean-k-mooney | that is form last month not form when we deiced not to block everything before christmas | |
| 17:40:24 | sean-k-mooney | a live snapshot will work | |
| 17:40:41 | sean-k-mooney | a cold snapshot might should too | |
| 17:40:53 | sean-k-mooney | since it stops the vm and then starts it after | |
| 17:41:37 | Sundar | A snapshot is going to save the mmap'd buffers in VM memory, and bring it back on restore. But the accelerator state would have changed in between as the VM continued op. So, on restore, the buffers in the VM will not match the accelerator contents. | |
| 17:42:07 | sean-k-mooney | Sundar: no it wont | |
| 17:42:21 | sean-k-mooney | a snappshot only makes a copy of the root disk | |
| 17:42:38 | sean-k-mooney | its not the same as a vm snapshot in virtual box or hyperv | |
| 17:42:44 | Sundar | I thought there's an option to do snapshot with RAM. | |
| 17:42:51 | sean-k-mooney | no | |
| 17:42:57 | sean-k-mooney | not in the libvirt dirver | |
| 17:43:10 | sean-k-mooney | you can do that with qemu but not via nova | |
| 17:43:13 | Sundar | libvirt allows it -- but apparently not the Nova libvirt driver? | |
| 17:43:27 | Sundar | I see | |
| 17:43:38 | sean-k-mooney | yes becasue we have no where to store the ram dump | |
| 17:44:31 | sean-k-mooney | we could in theory upload it as a second image to glace or something but snapshot in the nova api is just of the disk | |
| 17:46:00 | Sundar | Are you able to check snapshot with FPGA-attached VMs? I'll try too -- but what works in the lab may not always work in the field. | |
| 17:46:41 | sean-k-mooney | i have not deployed on real hardware so no. | |
| 17:48:40 | Sundar | I would say that it is safer to block something rather than speculatively allow it. Do you have users who strongly need snapshots with accelerators? | |
| 17:49:43 | sean-k-mooney | not specifically but i would not consider movingit form techpreview without it | |
| 17:50:10 | sean-k-mooney | in the curent patch there are no ways to do any move operation and you cant resuceu or snapshot the vm | |
| 17:50:28 | sean-k-mooney | so there is 0 way to do maintaince on the host or recover in the case of a vailure | |
| 17:50:33 | sean-k-mooney | even evacuate is blocked | |
| 17:50:58 | sean-k-mooney | as is adding or removeing volumes or interfaces | |
| 17:51:22 | sean-k-mooney | so with only the abilty to boot and delete a vm we would not support this in productin for customer to use | |
| 17:52:21 | sean-k-mooney | thats a downstrema desciion however. it wont affect if we support this in upstream | |
| 17:52:56 | Sundar | I see your concern. The real solution is to test these ops well with real accelerators and real-ish workloads (not sample bitstreams). I'll pass that on to my Cyborg/Intel colleagues. | |
| 17:53:22 | Sundar | Meanwhile, based on your input, I could remove the blocks on snapshot and rescue. Presumably backup too? | |
| 17:54:16 | sean-k-mooney | backup is basicaly a cron job in the nova code that calls snapshot and clean up old snapshots | |
| 17:54:27 | sean-k-mooney | so if snapshot work then back will | |
| 17:55:24 | sean-k-mooney | im not sure about rescue. it should work but i dont know if the acclerator will be aviable while its in resuce state | |
| 17:55:53 | sean-k-mooney | if the accelerator in not present in the vm during or after resuce has been completed then we might need to block it | |
| 17:56:32 | Sundar | Since the rescue brings in a different disk image, which may not have the right device drivers, it probably will not. The unrescue to the previous image "should" work. | |
| 17:57:04 | sean-k-mooney | oh the other hand not everyone would need the acclerator to be present during resuce so as long as we dont loose itafter unrescue we should be ok | |
| 17:57:23 | Sundar | Ok. So rescue is in. What about suspend? It doesn't work today because we don't have the code to handle accelerators in that path. | |
| 17:57:51 | sean-k-mooney | suspend wont work without a code change so ya that can remain blocked | |
| 17:58:31 | openstackgerrit | Mark Goddard proposed openstack/nova master: Fix PCI passthrough race on reschedule (claims) https://review.opendev.org/710847 | |
| 17:58:32 | openstackgerrit | Mark Goddard proposed openstack/nova master: Fix PCI passthrough race on reschedule (refresh) https://review.opendev.org/710848 | |
| 17:59:23 | Sundar | sean-k-mooney: The other 2 ops mentioned in https://review.opendev.org/#/c/674726/3/nova/tests/functional/test_servers.py are rebuild and resize. | |
| 17:59:42 | dansmith | Sundar: sean-k-mooney just catching up, but if we can't support snapshot, that's a huge problem IMHO (although I like sean-k-mooney think it's likely not a problem) | |
| 18:00:19 | dansmith | if rescue doesn't include the accelerator I don't think we should block it | |