| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-09 | |||
| 11:45:58 | sean-k-mooney | it kind of looks like a rabbitmq issue | |
| 11:54:45 | slaweq | sean-k-mooney: rabbitmq timeout happend first time around 19:57 in nova-compute on subnode-2 | |
| 11:55:05 | slaweq | in rabbitmq logs it seems that during this time there were only closing connection logs: https://02475c780c6fc32e71dc-f63b2c309fc0040fbb4a377b77794f40.ssl.cf1.rackcdn.com/697035/1/check/neutron-grenade-dvr-multinode/baba7e2/logs/rabbitmq/rabbit%40ubuntu-bionic-ovh-gra1-0013217908.txt.gz | |
| 11:57:05 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072 | |
| 13:07:15 | openstackgerrit | Merged openstack/nova master: Imported Translations from Zanata https://review.opendev.org/694717 | |
| 13:18:37 | mgariepy | anyone had issue with ephemeral storage when upgrading to 18.04 ? | |
| 13:19:41 | mgariepy | the _base image for the ephemeral part is formated with differents options, so the backing file for the disk.eph0 is not quite right and will not work for the vm being migrated. | |
| 13:29:10 | sean-k-mooney | dansmith: i spent some time over the weekend playing with the pci endpoint test driver and the netdevsim module. | |
| 13:29:39 | sean-k-mooney | dansmith: they will not allow use to fake pci devices unfortunetly | |
| 13:30:10 | sean-k-mooney | dansmith: the endpoint driver need a pci endpoint contoler to be present and there is not software implematnion of that | |
| 13:31:23 | sean-k-mooney | and the netdevsim module still has the limitation that it just simulates the netdevs not the pci endpoint. so even though you can create PFs and VFs it does not create pcie endpoint just the netdevs | |
| 13:32:06 | sean-k-mooney | so its the same as when i last looked at this in febuary | |
| 13:49:04 | huaqiang | sean-k-mooney: In Train release, you have reviewed my proposal for using PCPU and VCPU in same instance, can you review the updated Ussuri version once you have spare time? | |
| 13:49:46 | huaqiang | sean-k-mooney: the spec's link is https://review.opendev.org/#/c/668656/ | |
| 13:49:50 | sean-k-mooney | this one https://review.opendev.org/#/c/668656/ | |
| 13:49:52 | sean-k-mooney | :) | |
| 13:49:54 | kashyap | melwitt: Hiya, will look | |
| 13:50:09 | huaqiang | thanks | |
| 13:50:19 | sean-k-mooney | huaqiang: ill finish the email im typeing and review it then | |
| 13:51:02 | huaqiang | I prepared the POC code, under topic: bp/mixed-cpu-instance-set4 | |
| 13:51:18 | huaqiang | not good enough but it works somehow | |
| 13:51:59 | sean-k-mooney | huaqiang: ideally the topic shoudl match the bluepinrt name so it should be bp/use-pcpu-vcpu-in-one-instance | |
| 13:52:37 | huaqiang | sean-k-mooney: I'll make the name be consistent in next update | |
| 13:53:54 | kashyap | melwitt: Ah, never mind, I see it's merged - the CPU comaprison check on AArch64 | |
| 13:53:54 | sean-k-mooney | ok i just helps when trying to find all the related patches if they have the same topic as the spec and it matchs the blueprint | |
| 13:55:30 | huaqiang | I saw that, appreciate that. | |
| 14:04:15 | aarents | hi, | |
| 14:04:26 | aarents | mriedem: I don't know if you got some news from Matt Booth about https://review.opendev.org/#/c/696084/ , he's probably off or busy? | |
| 14:05:52 | kashyap | aarents: He normally goes by 'mdbooth' in this channel | |
| 14:08:27 | mdbooth | kashyap: Did you look at the arguments involved https://review.opendev.org/#/c/696084/2/nova/virt/libvirt/imagebackend.py ? | |
| 14:08:37 | mdbooth | Is that guaranteed to flatten the qcow2? | |
| 14:09:01 | kashyap | mdbooth: Afraid, not yet; was addressing something else. /me goes to look... | |
| 14:12:04 | kashyap | mdbooth: To flatten a chain *offline* shouldn't one require `qemu-img commit`? (Online is 'block-commit') | |
| 14:12:58 | mdbooth | kashyap: Right. I haven't checked the exact args used there, or refreshed my memory on qcow2 flattening, but I seemed to recall there was more involved | |
| 14:13:41 | kashyap | mdbooth: Definitely commit is required. | |
| 14:13:51 | kashyap | So says my 2012 "handout" even :D -- https://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html | |
| 14:14:07 | kashyap | If you have: [base] <-- [overlay1] | |
| 14:14:28 | kashyap | To "flatten" it, qemu-img commit sn2.qcow2 | |
| 14:14:35 | kashyap | [Err, bad copy/paste.] | |
| 14:14:47 | kashyap | To "flatten" it, `qemu-img commit overlay1.qcow2`. | |
| 14:15:19 | kashyap | [If you have more than two files in a chain U2014which is it in our case — | |
| 14:15:51 | kashyap | [If you have more than two files in a chain — which is it in our case — then you'd also have to update the backing file pointer.] | |
| 14:16:09 | lyarwood | I don't think you need to do that anymore kashyap, I've only ever seen just a normal qcow2 to qcow2 convert used to flatten qcow2 disks. | |
| 14:16:26 | kashyap | lyarwood: Sorry, what is not required anymore? | |
| 14:16:34 | kashyap | lyarwood: Ah, the 'commit'? | |
| 14:17:05 | kashyap | Right, 'qemu-img conver' is another way. Which begs the question which method is preferred over the other, and why | |
| 14:17:33 | lyarwood | yeah, http://paste.openstack.org/show/787313/ | |
| 14:19:32 | kashyap | lyarwood: I _think_ the advantage of using 'convert' is that it retains some qcow2 properties... | |
| 14:26:02 | lyarwood | kashyap: yeah I think so, I've commented on the change anyway, for qcow2 I think we need to rebase on the original cached image otherwise each unshelve is going to eat up a whole load of disk with each call to flatten. | |
| 14:26:19 | lyarwood | mdbooth: ^ not sure if you agree | |
| 14:26:53 | mriedem | gibi_off: i replied to your questions in https://review.opendev.org/#/c/637070/ - let me know if you need anything else | |
| 14:27:20 | mriedem | elod: do you know when gibi is back? | |
| 14:28:07 | kashyap | lyarwood: There are _three_ ways in total Including 'rebase' -- I was remembering it only changes the backing file pointer. | |
| 14:28:13 | elod | mriedem: wednesday, if I'm not mistaken | |
| 14:28:17 | kashyap | lyarwood: Will update the change with the three ways, and pros/cons | |
| 14:28:21 | mriedem | elod: ok thanks | |
| 14:30:45 | lyarwood | kashyap: ah I wasn't aware that you could rebase disks between backing files, that would be super useful here. | |
| 14:30:51 | lyarwood | kashyap: thanks! | |
| 14:31:37 | kashyap | lyarwood: Yes, indeed. By default 'qemu-img rebase' does a "real rebase" option. And 'qemu-img rebase -U' -- "unsafe" -- will only update the backing file | |
| 14:31:57 | kashyap | The man page explains it. /me recalls documenting that part of 'qemu-img' many moons ago :-) | |
| 14:32:14 | kashyap | (Read the Safe mode vs. Unsafe mode discussion) | |
| 14:36:11 | aarents | mdbooth: kashyap Hi, basicly by using convert, I reused stuff done when we extract a snapshot to glance (we need to flaten before upload) | |
| 14:42:43 | kashyap | aarents: Hi, there are a couple of trade-offs here, based on the method we're using | |
| 14:43:38 | kashyap | 'convert' has the advantage of also handling sparsification (as guessed earlier); but has the disadvantage of being relatively slow _and_ requires double the space | |
| 14:44:52 | kashyap | aarents: Slow because, 'convert' copies both base and overlay into a new image; and thus double the space. (Instead of copying only into base or the overlay, in case of 'commit' or 'rebase') | |
| 14:45:23 | kashyap | aarents: Writing a further comment in the change, once we have the options laid out, then we can make a decision | |
| 15:00:21 | aarents | kashyap: ok great | |
| 15:18:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Follow up to I5b9d41ef34385689d8da9b3962a1eac759eddf6a https://review.opendev.org/698028 | |
| 15:18:36 | kashyap | mdbooth: aarents: lyarwood: Added comparison notes of the three possible ways we can take: https://review.opendev.org/#/c/696084/ | |
| 15:20:54 | efried | slaweq: catching up... | |
| 15:23:18 | efried | slaweq: Any time I see spurious grenade fails in the last couple months, I attribute it to oversubscribed CI nodes, per mriedem's "State of the Gate" thread, started here http://lists.openstack.org/pipermail/openstack-discuss/2019-October/thread.html#10484 and continued here http://lists.openstack.org/pipermail/openstack-discuss/2019-November/thread.html#10502 | |
| 15:24:48 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 15:24:48 | efried | slaweq: and per the first note in that thread, yes, the bug you identified (bug 1844929) is the one we're "tracking" the issue with. | |
| 15:26:10 | efried | ...and the right solution is to get the CI providers to tweak their environments accordingly. We would rather have lower job throughput and lower failure rates. | |
| 15:26:18 | efried | But so far there has been no reaction from them. | |
| 15:26:45 | mriedem | note that for that particular bug the vast majority of fails are on ovh nodes | |
| 15:27:33 | mriedem | i don't know why it's mostly on grenade jobs | |
| 15:27:58 | mriedem | must have something to do with restarting mysql a few times, but idk | |
| 15:29:00 | mriedem | note that it also started with train, i don't know why though | |
| 15:29:12 | mriedem | tl;dr i don't really know much of anything | |
| 15:33:26 | sean-k-mooney | do we also restart rabbitmq during the grenade upgrade. i assume so but i have not checked | |
| 15:33:58 | sean-k-mooney | it is porably restarted at least once by devstack on the second stacking | |
| 15:35:11 | sean-k-mooney | i noticed that the subnode hand messaging timeouts in both the n-cpu and q-agt services | |
| 15:35:57 | sean-k-mooney | so it looked like we lost messages or something | |
| 15:45:14 | slaweq | mriedem: efried thx for confirmation that this is the bug which we hit most of the times now | |
| 15:45:18 | efried | dansmith: it would seem that the 422 event code is masked when there's only one event (as is the case with cyborg) | |
| 15:45:25 | slaweq | sean-k-mooney: and thx for info about rabbitmq too | |
| 15:45:36 | efried | slaweq: It would help to get some more voices complaining at the node providers. | |
| 15:45:54 | dansmith | efried: really? I didn't see that condition | |
| 15:45:58 | efried | dansmith: when all events are dropped, the handler turns the whole thing into a 404 | |
| 15:46:06 | efried | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/server_external_events.py#L146 | |
| 15:46:41 | efried | which is arguably a bug | |
| 15:46:44 | dansmith | efried: ah, yeah, that code isn't really right.. "no instances found for any event" isn't true, just no hosts.. | |
| 15:46:46 | dansmith | yeah | |
| 15:47:05 | efried | to "fix" the bug would be a microversion? | |
| 15:47:07 | dansmith | unless mriedem feels that's a microversion problem, I can change that | |
| 15:47:31 | dansmith | I dunno, it's correcting the error code, which I thought was allowable in some situations | |
| 15:47:51 | efried | what would you correct it to? 207? | |
| 15:48:26 | efried | Meaning there's actually no way the API ever returns an error. | |
| 15:48:28 | dansmith | that's what we return per-event in that case right? | |