| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-09 | |||
| 05:30:04 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Define Cyborg ARQ binding notification event. https://review.opendev.org/692707 | |
| 05:30:04 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Create and bind Cyborg ARQs. https://review.opendev.org/631244 | |
| 05:30:05 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Compose accelerator PCI devices into VM's domain XML. https://review.opendev.org/631245 | |
| 05:30:05 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted. https://review.opendev.org/673735 | |
| 05:30:06 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Enable hard reboot with accelerators. https://review.opendev.org/697940 | |
| 05:30:06 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999 | |
| 06:44:43 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.opendev.org/694717 | |
| 09:16:29 | openstackgerrit | Brin Zhang proposed openstack/nova master: PoC: Support re-configure the delete_on_termination in server https://review.opendev.org/693828 | |
| 11:18:35 | slaweq | efried: hi | |
| 11:18:59 | slaweq | efried: recently I saw quite often grenade jobs failing with errors like https://02475c780c6fc32e71dc-f63b2c309fc0040fbb4a377b77794f40.ssl.cf1.rackcdn.com/697035/1/check/neutron-grenade-dvr-multinode/baba7e2/logs/testr_results.html.gz | |
| 11:19:24 | slaweq | basically it is "No valid host was found. There are not enough hosts available.'" error from nova in some tests and tests are failing | |
| 11:19:51 | slaweq | efried: did You see something similar? and is there any bug reported for that or should I open new one? | |
| 11:23:03 | slaweq | efried: and in the scheduler log I see something like here: https://zuul.opendev.org/t/openstack/build/baba7e2f78994deabbc3230b3f9acc80/log/logs/screen-n-sch.txt.gz#2505 | |
| 11:23:43 | slaweq | and before that there is log like: https://zuul.opendev.org/t/openstack/build/baba7e2f78994deabbc3230b3f9acc80/log/logs/screen-n-sch.txt.gz#2501 | |
| 11:23:54 | slaweq | "Timed out waiting for response from cell 258a967d-07ef-43a6-b81e-2ac433a583ef" | |
| 11:24:29 | slaweq | efried: so now I think it's the same issue as https://bugs.launchpad.net/nova/+bug/1844929 - is that correct? | |
| 11:24:29 | openstack | Launchpad bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] | |
| 11:34:27 | sean-k-mooney | the availabliy zone filter is a red herring that iteration started with 0 hosts https://zuul.opendev.org/t/openstack/build/baba7e2f78994deabbc3230b3f9acc80/log/logs/screen-n-sch.txt.gz#2503 | |
| 11:41:41 | sean-k-mooney | slaweq: we do see the same messaging timeout in the n-cpu log on the sub node https://02475c780c6fc32e71dc-f63b2c309fc0040fbb4a377b77794f40.ssl.cf1.rackcdn.com/697035/1/check/neutron-grenade-dvr-multinode/baba7e2/logs/subnode-2/screen-n-cpu.txt.gz | |
| 11:44:14 | sean-k-mooney | slaweq: im also seeing similar error in the agent log | |
| 11:45:58 | sean-k-mooney | it kind of looks like a rabbitmq issue | |
| 11:54:45 | slaweq | sean-k-mooney: rabbitmq timeout happend first time around 19:57 in nova-compute on subnode-2 | |
| 11:55:05 | slaweq | in rabbitmq logs it seems that during this time there were only closing connection logs: https://02475c780c6fc32e71dc-f63b2c309fc0040fbb4a377b77794f40.ssl.cf1.rackcdn.com/697035/1/check/neutron-grenade-dvr-multinode/baba7e2/logs/rabbitmq/rabbit%40ubuntu-bionic-ovh-gra1-0013217908.txt.gz | |
| 11:57:05 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072 | |
| 13:07:15 | openstackgerrit | Merged openstack/nova master: Imported Translations from Zanata https://review.opendev.org/694717 | |
| 13:18:37 | mgariepy | anyone had issue with ephemeral storage when upgrading to 18.04 ? | |
| 13:19:41 | mgariepy | the _base image for the ephemeral part is formated with differents options, so the backing file for the disk.eph0 is not quite right and will not work for the vm being migrated. | |
| 13:29:10 | sean-k-mooney | dansmith: i spent some time over the weekend playing with the pci endpoint test driver and the netdevsim module. | |
| 13:29:39 | sean-k-mooney | dansmith: they will not allow use to fake pci devices unfortunetly | |
| 13:30:10 | sean-k-mooney | dansmith: the endpoint driver need a pci endpoint contoler to be present and there is not software implematnion of that | |
| 13:31:23 | sean-k-mooney | and the netdevsim module still has the limitation that it just simulates the netdevs not the pci endpoint. so even though you can create PFs and VFs it does not create pcie endpoint just the netdevs | |
| 13:32:06 | sean-k-mooney | so its the same as when i last looked at this in febuary | |
| 13:49:04 | huaqiang | sean-k-mooney: In Train release, you have reviewed my proposal for using PCPU and VCPU in same instance, can you review the updated Ussuri version once you have spare time? | |
| 13:49:46 | huaqiang | sean-k-mooney: the spec's link is https://review.opendev.org/#/c/668656/ | |
| 13:49:50 | sean-k-mooney | this one https://review.opendev.org/#/c/668656/ | |
| 13:49:52 | sean-k-mooney | :) | |
| 13:49:54 | kashyap | melwitt: Hiya, will look | |
| 13:50:09 | huaqiang | thanks | |
| 13:50:19 | sean-k-mooney | huaqiang: ill finish the email im typeing and review it then | |
| 13:51:02 | huaqiang | I prepared the POC code, under topic: bp/mixed-cpu-instance-set4 | |
| 13:51:18 | huaqiang | not good enough but it works somehow | |
| 13:51:59 | sean-k-mooney | huaqiang: ideally the topic shoudl match the bluepinrt name so it should be bp/use-pcpu-vcpu-in-one-instance | |
| 13:52:37 | huaqiang | sean-k-mooney: I'll make the name be consistent in next update | |
| 13:53:54 | sean-k-mooney | ok i just helps when trying to find all the related patches if they have the same topic as the spec and it matchs the blueprint | |
| 13:53:54 | kashyap | melwitt: Ah, never mind, I see it's merged - the CPU comaprison check on AArch64 | |
| 13:55:30 | huaqiang | I saw that, appreciate that. | |
| 14:04:15 | aarents | hi, | |
| 14:04:26 | aarents | mriedem: I don't know if you got some news from Matt Booth about https://review.opendev.org/#/c/696084/ , he's probably off or busy? | |
| 14:05:52 | kashyap | aarents: He normally goes by 'mdbooth' in this channel | |
| 14:08:27 | mdbooth | kashyap: Did you look at the arguments involved https://review.opendev.org/#/c/696084/2/nova/virt/libvirt/imagebackend.py ? | |
| 14:08:37 | mdbooth | Is that guaranteed to flatten the qcow2? | |
| 14:09:01 | kashyap | mdbooth: Afraid, not yet; was addressing something else. /me goes to look... | |
| 14:12:04 | kashyap | mdbooth: To flatten a chain *offline* shouldn't one require `qemu-img commit`? (Online is 'block-commit') | |
| 14:12:58 | mdbooth | kashyap: Right. I haven't checked the exact args used there, or refreshed my memory on qcow2 flattening, but I seemed to recall there was more involved | |
| 14:13:41 | kashyap | mdbooth: Definitely commit is required. | |
| 14:13:51 | kashyap | So says my 2012 "handout" even :D -- https://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html | |
| 14:14:07 | kashyap | If you have: [base] <-- [overlay1] | |
| 14:14:28 | kashyap | To "flatten" it, qemu-img commit sn2.qcow2 | |
| 14:14:35 | kashyap | [Err, bad copy/paste.] | |
| 14:14:47 | kashyap | To "flatten" it, `qemu-img commit overlay1.qcow2`. | |
| 14:15:19 | kashyap | [If you have more than two files in a chain U2014which is it in our case — | |
| 14:15:51 | kashyap | [If you have more than two files in a chain — which is it in our case — then you'd also have to update the backing file pointer.] | |
| 14:16:09 | lyarwood | I don't think you need to do that anymore kashyap, I've only ever seen just a normal qcow2 to qcow2 convert used to flatten qcow2 disks. | |
| 14:16:26 | kashyap | lyarwood: Sorry, what is not required anymore? | |
| 14:16:34 | kashyap | lyarwood: Ah, the 'commit'? | |
| 14:17:05 | kashyap | Right, 'qemu-img conver' is another way. Which begs the question which method is preferred over the other, and why | |
| 14:17:33 | lyarwood | yeah, http://paste.openstack.org/show/787313/ | |
| 14:19:32 | kashyap | lyarwood: I _think_ the advantage of using 'convert' is that it retains some qcow2 properties... | |
| 14:26:02 | lyarwood | kashyap: yeah I think so, I've commented on the change anyway, for qcow2 I think we need to rebase on the original cached image otherwise each unshelve is going to eat up a whole load of disk with each call to flatten. | |
| 14:26:19 | lyarwood | mdbooth: ^ not sure if you agree | |
| 14:26:53 | mriedem | gibi_off: i replied to your questions in https://review.opendev.org/#/c/637070/ - let me know if you need anything else | |
| 14:27:20 | mriedem | elod: do you know when gibi is back? | |
| 14:28:07 | kashyap | lyarwood: There are _three_ ways in total Including 'rebase' -- I was remembering it only changes the backing file pointer. | |
| 14:28:13 | elod | mriedem: wednesday, if I'm not mistaken | |
| 14:28:17 | kashyap | lyarwood: Will update the change with the three ways, and pros/cons | |
| 14:28:21 | mriedem | elod: ok thanks | |
| 14:30:45 | lyarwood | kashyap: ah I wasn't aware that you could rebase disks between backing files, that would be super useful here. | |
| 14:30:51 | lyarwood | kashyap: thanks! | |
| 14:31:37 | kashyap | lyarwood: Yes, indeed. By default 'qemu-img rebase' does a "real rebase" option. And 'qemu-img rebase -U' -- "unsafe" -- will only update the backing file | |
| 14:31:57 | kashyap | The man page explains it. /me recalls documenting that part of 'qemu-img' many moons ago :-) | |
| 14:32:14 | kashyap | (Read the Safe mode vs. Unsafe mode discussion) | |
| 14:36:11 | aarents | mdbooth: kashyap Hi, basicly by using convert, I reused stuff done when we extract a snapshot to glance (we need to flaten before upload) | |
| 14:42:43 | kashyap | aarents: Hi, there are a couple of trade-offs here, based on the method we're using | |
| 14:43:38 | kashyap | 'convert' has the advantage of also handling sparsification (as guessed earlier); but has the disadvantage of being relatively slow _and_ requires double the space | |
| 14:44:52 | kashyap | aarents: Slow because, 'convert' copies both base and overlay into a new image; and thus double the space. (Instead of copying only into base or the overlay, in case of 'commit' or 'rebase') | |
| 14:45:23 | kashyap | aarents: Writing a further comment in the change, once we have the options laid out, then we can make a decision | |
| 15:00:21 | aarents | kashyap: ok great | |
| 15:18:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Follow up to I5b9d41ef34385689d8da9b3962a1eac759eddf6a https://review.opendev.org/698028 | |
| 15:18:36 | kashyap | mdbooth: aarents: lyarwood: Added comparison notes of the three possible ways we can take: https://review.opendev.org/#/c/696084/ | |
| 15:20:54 | efried | slaweq: catching up... | |
| 15:23:18 | efried | slaweq: Any time I see spurious grenade fails in the last couple months, I attribute it to oversubscribed CI nodes, per mriedem's "State of the Gate" thread, started here http://lists.openstack.org/pipermail/openstack-discuss/2019-October/thread.html#10484 and continued here http://lists.openstack.org/pipermail/openstack-discuss/2019-November/thread.html#10502 | |
| 15:24:48 | efried | slaweq: and per the first note in that thread, yes, the bug you identified (bug 1844929) is the one we're "tracking" the issue with. | |
| 15:24:48 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 15:26:10 | efried | ...and the right solution is to get the CI providers to tweak their environments accordingly. We would rather have lower job throughput and lower failure rates. | |
| 15:26:18 | efried | But so far there has been no reaction from them. | |
| 15:26:45 | mriedem | note that for that particular bug the vast majority of fails are on ovh nodes | |
| 15:27:33 | mriedem | i don't know why it's mostly on grenade jobs | |
| 15:27:58 | mriedem | must have something to do with restarting mysql a few times, but idk | |
| 15:29:00 | mriedem | note that it also started with train, i don't know why though | |
| 15:29:12 | mriedem | tl;dr i don't really know much of anything | |