| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-22 | |||
| 16:20:09 | melwitt | hm k | |
| 16:20:16 | bauzas | JayF: are you around ? | |
| 16:20:59 | JayF | What's up? | |
| 16:21:21 | bauzas | JayF: about https://review.opendev.org/c/openstack/nova/+/903915/ I wonder if you could provide a new revision due to sean-k-mooney's -1 ? | |
| 16:21:34 | bauzas | also, do we now have a Tempest job testing it ? | |
| 16:22:24 | JayF | Sean and I talked in depth about a testing plan earlier this week, and today and tomorrow is the time I set aside to implement it. At which point I'll re-stack all the sharding changes, make sure they pass CI, and do some manual testing | |
| 16:23:08 | JayF | We have implemented significant sharding testing in ironic for ensuring the API works properly, the second step I'm working on today is the scenario tests | |
| 16:23:09 | sean-k-mooney | cool just an fyi tomorrow is a "rechage day" at redhat meaning none of us will be here tomorow | |
| 16:23:14 | sean-k-mooney | but we can take a look on monday | |
| 16:23:35 | JayF | Yeah I think I have all the information I need based on our conversation earlier, I just need to not have my day stolen by a thousand tiny conversations | |
| 16:32:37 | melwitt | sean-k-mooney: the new_websocket_client method is what gets called when a new request to the proxy comes in, so I also dunno why that would need to be called to close a connection ... ? | |
| 16:37:17 | sean-k-mooney | auniyal: can you try https://review.opendev.org/c/openstack/nova/+/901824/18/nova/console/websocketproxy.py#154 | |
| 16:46:38 | auniyal | sean-k-mooney, ack thanks | |
| 16:48:51 | auniyal | sean-k-mooney, can we raise an Exception as invalid target token, inside | |
| 16:49:00 | auniyal | reason I am saying is https://github.com/openstack/nova/blob/master/nova/console/websocketproxy.py#L265 | |
| 16:50:59 | sean-k-mooney | that is coming from self.do_proxy | |
| 16:51:01 | sean-k-mooney | right | |
| 16:51:28 | auniyal | yes, so there some exception will be raised, like something happend in network or at client | |
| 16:52:00 | sean-k-mooney | so if we think about the reason why that might happen | |
| 16:52:22 | sean-k-mooney | the only reason i can think of is the client already disconnect before the time expired | |
| 16:52:43 | sean-k-mooney | in which case i dont think we should be raising an excption as that is an expected scenairo | |
| 16:53:06 | sean-k-mooney | so im not convinced raising an excption is correct | |
| 16:53:14 | auniyal | ack, | |
| 16:53:20 | sean-k-mooney | a debug level log woudl be fine | |
| 16:53:32 | sean-k-mooney | but do you have any other edge case in mind | |
| 16:53:51 | auniyal | okay, also if something really happend at client side https://review.opendev.org/c/openstack/nova/+/901824/18/nova/console/websocketproxy.py#294 | |
| 16:54:35 | auniyal | no, so as you said, everytime we come to close connection we will have valid tsock | |
| 16:55:03 | auniyal | ack, will remove if-else then, thanks | |
| 16:56:22 | sean-k-mooney | you have already protected agasint the socket being cloased with the OSError and the fileno being -1 | |
| 16:56:37 | sean-k-mooney | so i htink you have handled all the edgecase that are required there | |
| 16:59:06 | auniyal | ack, I'll just run tests locally and respin | |
| 16:59:55 | opendevreview | sean mooney proposed openstack/nova master: [S-RBAC] adapt nova-next for port's binding:profile field change https://review.opendev.org/c/openstack/nova/+/909859 | |
| 17:06:43 | sean-k-mooney | melwitt: by the way using blockcopy instead of blockrebase is a nice find | |
| 17:07:04 | sean-k-mooney | melwitt: it remiened me of something for future us | |
| 17:07:31 | sean-k-mooney | we currenly have some legacy code in hwo we do live snapshots to workaround some bugs with older version fo qemu/libvirt | |
| 17:07:42 | sean-k-mooney | my understanding is those issues have been fixed | |
| 17:07:57 | sean-k-mooney | so we may be able to simply that code alot in the future | |
| 17:08:08 | sean-k-mooney | by removing the legacy workaround | |
| 17:08:16 | melwitt | sean-k-mooney: yeah, I couldn't stop thinking about live snapshot not working so I asked in #virt and they advised to use blockcopy in order to provide encryption options | |
| 17:08:39 | melwitt | yeah, I think we likely could | |
| 17:09:23 | sean-k-mooney | we do this complciated dance where we start a rebase or copy jobs afte we copy a base iamge then abort it to do the live snapshot | |
| 17:09:55 | sean-k-mooney | i.e. we copy the backing file, start a jobs to sysnc the chagnes, briefly freese the guest file system and then abort | |
| 17:10:30 | sean-k-mooney | this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3412 | |
| 17:11:21 | melwitt | right yeah | |
| 17:11:50 | sean-k-mooney | so i need to do git balame locally because github hate doing it on that file | |
| 17:12:04 | sean-k-mooney | but that is a hack and should not be required | |
| 17:12:27 | bauzas | JayF: (sorry, forgot to see your reply) sure, I'll lookup then your patch tomorrow (even if at RH, I'll work on tomorrow morning) | |
| 17:12:52 | JayF | Today and tomorrow :) So look tomorrow but don't expect until Monday | |
| 17:14:11 | opendevreview | Amit Uniyal proposed openstack/nova master: enforce remote console shutdown https://review.opendev.org/c/openstack/nova/+/901824 | |
| 17:16:34 | sean-k-mooney | melwitt: so this dates form when we first added live snapshot https://github.com/uggla/nova/commit/46de2d1e2d0abd6fdcd4da13facaf3225c721f5e | |
| 17:17:52 | melwitt | quite a bit ago :) | |
| 17:18:37 | sean-k-mooney | yep so the simpeler ways of doign this were not "bug-free" in qemu 1.3 | |
| 17:19:07 | sean-k-mooney | but we havng used that in half a decade or more | |
| 17:21:00 | melwitt | yeah, agree it would be good to clean that up | |
| 17:25:13 | sean-k-mooney | based on https://libvirt.org/kbase/domainstatecapture.html we have a few options | |
| 17:27:16 | sean-k-mooney | i would hope we could use the direct backup functionaltiy or similar | |
| 17:28:22 | sean-k-mooney | something like this https://libvirt.org/formatbackup.html#examples | |
| 17:29:51 | melwitt | ah, ok | |
| 17:30:40 | sean-k-mooney | the thing is i dont know if any of that will work with encyption | |
| 17:31:30 | sean-k-mooney | if it does not virDomainSnapshotCreateXML may or we might just need to keep what we are doing | |
| 17:31:46 | sean-k-mooney | but it wooul dbe a good conversation to have with the virt folks about | |
| 17:32:04 | sean-k-mooney | "if we were ot do it from scratch today what would you recommend" | |
| 17:32:38 | melwitt | yeah. I think they mentioned the real snapshot API would be the better way to do it (with encryption) | |
| 17:33:48 | sean-k-mooney | this one https://libvirt.org/html/libvirt-libvirt-domain-snapshot.html#virDomainSnapshotCreateXML | |
| 17:34:02 | sean-k-mooney | ya qemu can aslo do memory snapshots and other fancy thigns via that | |
| 17:34:37 | sean-k-mooney | we can pass flags=VIR_DOMAIN_SNAPSHOT_CREATE_LIVE|VIR_DOMAIN_SNAPSHOT_CREATE_DISK_ONLY | |
| 17:35:22 | sean-k-mooney | and there is VIR_DOMAIN_SNAPSHOT_CREATE_REUSE_EXT | |
| 17:35:33 | sean-k-mooney | which alluse you ot resue a precreate file i think | |
| 17:36:05 | melwitt | yeah that's right | |
| 17:39:03 | sean-k-mooney | anway that a long way to say im +1 on the bockcopy usage and we may be able to simplfy later | |
| 17:40:35 | melwitt | ack ++ :) | |
| 19:22:07 | opendevreview | Dan Smith proposed openstack/nova master: Catch ImageNotFound on snapshot failure https://review.opendev.org/c/openstack/nova/+/905316 | |
| 19:22:08 | opendevreview | Dan Smith proposed openstack/nova master: DNM: Test glance new location api https://review.opendev.org/c/openstack/nova/+/891207 | |
| 19:22:08 | opendevreview | Dan Smith proposed openstack/nova master: Support glance's new location API https://review.opendev.org/c/openstack/nova/+/891036 | |
| 19:25:20 | melwitt | sean-k-mooney: nova-next is still unhappy :( https://zuul.opendev.org/t/openstack/build/3c2c7955a4b34112a377a857623d6a73/log/job-output.txt#37662 | |
| 19:26:36 | melwitt | guess we can't check for healed port allocations | |
| 19:32:05 | sean-k-mooney | really | |
| 19:33:01 | sean-k-mooney | or did i forget to delete on eof the checks | |
| 19:36:03 | sean-k-mooney | so tha tot me say that we have not configure bandwith qos properly | |
| 19:36:43 | sean-k-mooney | am i need ot go have somethign to eat but ok ill just drop all the port stuff | |
| 19:36:59 | sean-k-mooney | and if that does not work we can drop all the heal code | |
| 19:39:44 | mnaser | https://bugs.launchpad.net/nova/+bug/2052915 is affecting nova right now but i have no idea where to look at the refernece | |
| 19:40:36 | melwitt | sean-k-mooney: I'm not sure .. it's failing on "bandwidth_allocations=$(echo "$allocations" | grep NET_BW_EGR_KILOBIT_PER_SEC)" | |
| 19:40:36 | sean-k-mooney | that sounds like neutron broke upgrades | |
| 19:40:59 | sean-k-mooney | yep so if bandwith qos was working that allocation should exist | |
| 19:41:50 | melwitt | the weird thing is that it's supposed to "echo "Failed to heal port allocations."" if the above returned "" but it's not echoing that | |
| 19:42:22 | sean-k-mooney | yep we have allcoations but not for that type | |
| 19:42:32 | melwitt | oh, I see. ok | |
| 19:42:58 | sean-k-mooney | so what i ment by not configre correctly is i think we may not have enabel the qos extionion in the job | |
| 19:43:08 | sean-k-mooney | or somethign liek that | |
| 19:43:37 | opendevreview | sean mooney proposed openstack/nova master: [S-RBAC] adapt nova-next for port's binding:profile field change https://review.opendev.org/c/openstack/nova/+/909859 | |
| 19:44:24 | melwitt | doesn't the fact that it used to work before the service token change mean the extension is present? | |
| 19:44:52 | sean-k-mooney | i would have to read the code but not nessiarlay | |
| 19:45:19 | melwitt | ok | |
| 19:45:23 | sean-k-mooney | the allocation is create by nova but if we dont have teh qos extion is not loaded | |
| 19:45:43 | sean-k-mooney | what will happen is neutron will not make a placement resouce request in teh port | |
| 19:46:32 | sean-k-mooney | melwitt: the hook was not asserting the bandwith resouce request existed before we healed | |
| 19:46:54 | sean-k-mooney | or that the the port has port resouce requests | |
| 19:47:57 | melwitt | 😵💫 | |
| 19:55:18 | sean-k-mooney | it does look like the qos plugin was loaded https://7afe9236b7eaaf244ae9-d98f699786987cf0f7a232f67c9b09f7.ssl.cf2.rackcdn.com/909859/2/check/nova-next/3c2c795/controller/logs/etc/neutron/neutron_conf.txt | |