| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-02 | |||
| 14:46:35 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove QEMU_VERSION_REQ_SHARED https://review.opendev.org/710239 | |
| 14:46:35 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Use oslo.utils >= 4.1.0 to fetch format-specific image data https://review.opendev.org/710785 | |
| 14:47:34 | lyarwood | stephenfin: ^ vendor'd the oslo.utils change so we can backport this easier, addressed your feedback and fixed your follow up (you missed the removal of a test). | |
| 14:49:32 | stephenfin | ack | |
| 14:50:08 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM - Test TEMPEST_EXTEND_ATTACHED_ENCRYPTED_VOLUME https://review.opendev.org/707593 | |
| 14:50:19 | sean-k-mooney | lyarwood: looking at your q35 change i dont think the test failture are vaild. i.e. if they are real failure i think its like an issue with how the test is written rather then a real issue | |
| 14:51:00 | lyarwood | sean-k-mooney: agreed, I had a quick look this morning but haven't had time to follow up as yet | |
| 14:51:32 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: libvirt: Provide the backing file format when creating qcow2 disks https://review.opendev.org/710788 | |
| 14:51:44 | sean-k-mooney | lyarwood: looking at the failure "tempest.lib.exceptions.SSHExecCommandFailed: Command 'set -eu -o pipefail; PATH=$PATH:/sbin:/usr/sbin; blkid | grep -i config-2', exit status: 1, stderr:" | |
| 14:52:04 | sean-k-mooney | i think its looking for the config drive? | |
| 14:52:24 | sean-k-mooney | and for q35 its attached vai sata instead of ide right | |
| 14:52:40 | lyarwood | it's looking for a tagged disk | |
| 14:52:41 | sean-k-mooney | so it might be reported differently in blkid | |
| 14:52:45 | sean-k-mooney | ah ok | |
| 14:52:50 | lyarwood | yeah I think the string might be slightly different | |
| 14:53:04 | sean-k-mooney | ya same effect i guess | |
| 14:53:05 | lyarwood | actually is the tag even there for SCSI | |
| 14:53:18 | lyarwood | you'd likely need to probe /sys | |
| 14:53:31 | sean-k-mooney | im not sure | |
| 14:53:39 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: libvirt: Provide the backing file format when creating qcow2 disks https://review.opendev.org/710790 | |
| 15:22:34 | dansmith | sean-k-mooney: for your multinode cyborg tests, are you actually stacking another compute-and-cyborg-only node? I was thinking maybe I could try to hack that into place by just running another copy of compute with a hostname override in conf | |
| 15:23:09 | sean-k-mooney | yes im sinning up two vms 1 is an all in one node and the other is a pure compute | |
| 15:23:37 | sean-k-mooney | by pure compute i mean it just running the nova/neutron/cinder/cyborg compute agents | |
| 15:23:57 | sean-k-mooney | that is what the ci job is doing too | |
| 15:26:08 | dansmith | yeah, okay, just thinking about maybe some lighter way to do that | |
| 15:26:13 | sean-k-mooney | dansmith: this is more or less the local.confs im using http://paste.openstack.org/show/789773/ | |
| 15:26:36 | sean-k-mooney | dansmith: ya i have 32G of ram on my laptop so 2 8G vms is fine | |
| 15:26:45 | sean-k-mooney | the config trick might work | |
| 15:26:53 | sean-k-mooney | but it just felt like more effort | |
| 15:26:55 | dansmith | yeah, ram isn't the problem | |
| 15:26:57 | dansmith | could be | |
| 15:29:33 | sean-k-mooney | do you want me to kick off a ci run with the latest set of patches sundar pushed | |
| 15:30:35 | sean-k-mooney | i basically just need to recheck https://review.opendev.org/#/c/709641/ or update the depends-on url to point to the latest patch in the cain | |
| 15:31:11 | dansmith | sean-k-mooney: yeah, would be good I think | |
| 15:31:14 | sean-k-mooney | dansmith: removing the singelton seemed to resolve all the auth issues | |
| 15:31:17 | dansmith | I'm working through those now | |
| 15:31:21 | dansmith | sean-k-mooney: nice :) | |
| 15:32:14 | sean-k-mooney | ok ill point it to pull in the last funcitonal patch in the cain which is the one that block unsupported opperation and we can get the job to assert they all fail/are rejected as we expect | |
| 15:35:32 | sean-k-mooney | cool thats running we will know in about an hour | |
| 15:35:44 | dansmith | schweet | |
| 15:40:09 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-evacuate https://review.opendev.org/708235 | |
| 16:17:11 | openstackgerrit | John Garbutt proposed openstack/nova master: Add a tests to check when legacy access is removed https://review.opendev.org/710813 | |
| 16:25:06 | openstackgerrit | John Garbutt proposed openstack/nova master: Add a tests to check when legacy access is removed https://review.opendev.org/710813 | |
| 16:29:50 | openstackgerrit | Merged openstack/nova master: Add test coverage of existing os-agents policies https://review.opendev.org/701644 | |
| 16:58:09 | sean-k-mooney | dansmith: the test run is still executing but i am seeing some failes as expected. however the error code is 500 which seams incorrect | |
| 16:58:16 | sean-k-mooney | Body: b'{"computeFault": {"code": 500, "message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\\n | |
| 16:58:18 | sean-k-mooney | \'nova.exception.ForbiddenWithAccelerators\'>"}}' | |
| 16:58:33 | sean-k-mooney | it should be a 4XX error correct | |
| 16:58:38 | dansmith | ah yeah | |
| 16:58:58 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: WIP: Hey let's support routed networks y'all! https://review.opendev.org/656885 | |
| 17:00:05 | sean-k-mooney | it looks like it should be a 403 here which i dont think is correct either https://review.opendev.org/#/c/674726/3/nova/exception.py | |
| 17:00:21 | sean-k-mooney | ill wait till the jobs completes and check the api log | |
| 17:00:29 | sean-k-mooney | the 500 might be a tempest error code | |
| 17:00:56 | sean-k-mooney | its hard to tell just form the job console | |
| 17:16:48 | Sundar | sean-k-mooney: The error response comes from https://review.opendev.org/#/c/674726/3/nova/api/openstack/compute/servers.py@758 . I see other similar operations grouped in that segment. | |
| 17:17:25 | sean-k-mooney | the job just finished https://zuul.opendev.org/t/openstack/build/0f144bf83a614d31938d353a1bb05935 | |
| 17:17:37 | sean-k-mooney | im going to confirm if the 500 is from tempest or nova | |
| 17:17:45 | Sundar | I have tested it from the 'openstack server ...' CLI, so I don;t think it is tempest | |
| 17:18:26 | sean-k-mooney | well the "computefault" bit makes me think its from tempest | |
| 17:19:12 | Sundar | You would see a 500 error if you do 'openstack server suspend foo'. | |
| 17:19:34 | sean-k-mooney | its not a server error so it should not be a 500 | |
| 17:19:46 | sean-k-mooney | its a client error because you requested an unsupported operation | |
| 17:20:16 | sean-k-mooney | so it s houl be either a 400 "bad request" 409 "conflict" or 405 "method not supported" | |
| 17:20:31 | sean-k-mooney | proably a 400 or 409 we dont use 405 as far as i know | |
| 17:23:28 | sean-k-mooney | Sundar: so yes we are raising the exeption but it is not being caught and its being converted into a 500 http://paste.openstack.org/show/790209/ | |
| 17:25:55 | openstackgerrit | Merged openstack/nova master: Define Cyborg ARQ binding notification event. https://review.opendev.org/692707 | |
| 17:26:23 | Sundar | sean-k-mooney: Hmm, let me see what's happening with https://review.opendev.org/#/c/674726/3/nova/api/openstack/compute/servers.py@758 | |
| 17:27:12 | sean-k-mooney | the block operation patch needs to be rewored to only block operation that dont work | |
| 17:28:38 | Sundar | sean-k-mooney: Unless I have proven or validated that an op works, I have blocked it. There are many ops that I have not validated, such as backup. Plus, there are grounds to suspect such ops won't always work, since there is state in the accelerator. | |
| 17:28:51 | sean-k-mooney | Sundar: yep which im hard -1 on | |
| 17:29:13 | sean-k-mooney | im ok with blocking ops we know wont work | |
| 17:29:28 | sean-k-mooney | im not ok with blocking ops just because we feel like it. | |
| 17:29:48 | Sundar | It is safer to block an op than let it loose and document that it is untested with accelerators. I thought we had discussed that. | |
| 17:30:23 | Sundar | Not 'we feel like it;, but whether it is tested and whether there are grounds to believe it will work dafely all the time | |
| 17:30:26 | sean-k-mooney | we agreed not to block the ops and just document them | |
| 17:30:28 | Sundar | *safely | |
| 17:30:41 | sean-k-mooney | that is why https://review.opendev.org/#/c/674726 was abandoned | |
| 17:30:55 | Sundar | Hmm, I recall otherwise. Let me look for past IRC discussions. | |
| 17:31:39 | sean-k-mooney | we can bring it back but i dont think its ok to block things like rebuild or volume based snapthot when we know they should work and they do work at least with the fake driver | |
| 17:39:22 | Sundar | sean-k-mooney: Here is our past discussion: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-02-07.log.html#t2020-02-07T19:17:13 . I see different opinions, but you still want the doc-only approach. But you do agree that people won't read the docs till things break | |
| 17:39:23 | Sundar | (http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-02-07.log.html#t2020-02-07T19:31:39) | |
| 17:39:54 | Sundar | Is it not safer to block an unproven op (but only for accelerators obviously)? | |
| 17:40:09 | Sundar | I am not si sure that snapshots will work | |
| 17:40:11 | sean-k-mooney | that is form last month not form when we deiced not to block everything before christmas | |
| 17:40:24 | sean-k-mooney | a live snapshot will work | |
| 17:40:41 | sean-k-mooney | a cold snapshot might should too | |
| 17:40:53 | sean-k-mooney | since it stops the vm and then starts it after | |
| 17:41:37 | Sundar | A snapshot is going to save the mmap'd buffers in VM memory, and bring it back on restore. But the accelerator state would have changed in between as the VM continued op. So, on restore, the buffers in the VM will not match the accelerator contents. | |
| 17:42:07 | sean-k-mooney | Sundar: no it wont | |
| 17:42:21 | sean-k-mooney | a snappshot only makes a copy of the root disk | |
| 17:42:38 | sean-k-mooney | its not the same as a vm snapshot in virtual box or hyperv | |
| 17:42:44 | Sundar | I thought there's an option to do snapshot with RAM. | |
| 17:42:51 | sean-k-mooney | no | |
| 17:42:57 | sean-k-mooney | not in the libvirt dirver | |
| 17:43:10 | sean-k-mooney | you can do that with qemu but not via nova | |
| 17:43:13 | Sundar | libvirt allows it -- but apparently not the Nova libvirt driver? | |
| 17:43:27 | Sundar | I see | |
| 17:43:38 | sean-k-mooney | yes becasue we have no where to store the ram dump | |
| 17:44:31 | sean-k-mooney | we could in theory upload it as a second image to glace or something but snapshot in the nova api is just of the disk | |
| 17:46:00 | Sundar | Are you able to check snapshot with FPGA-attached VMs? I'll try too -- but what works in the lab may not always work in the field. | |