Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-11
17:19:42 sean-k-mooney oh i see this is related to vtpm
17:24:20 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Resolve (most) flake8 3.x issues https://review.opendev.org/695732
17:24:21 openstackgerrit Stephen Finucane proposed openstack/nova master: Switch to flake8 3.x https://review.opendev.org/695733
17:31:32 mriedem wtf, so back on dec 5 i had a passing run of nova-multi-cell with migration tests enabled. now since the 9th with a new run all migration tests are failing because once the confirmed resized server is active again the api is saying the flavor is the old id even though i can see in the conductor logs where right before that the instance has the correct flavor
17:31:50 mriedem the api is pulling from the correct cell
17:32:50 openstackgerrit Stephen Finucane proposed openstack/nova master: Add resource provider allocation unset example to troubleshooting doc https://review.opendev.org/696582
17:33:12 stephenfin mriedem: Revert that if you don't like what I've done. It was just easier do it than explain what I wanted in a comment ^
17:33:23 stephenfin efried: you asked about that earlier ^
17:34:23 mriedem ok
17:40:35 openstackgerrit Merged openstack/nova master: Use provider mappings from Placement (mostly) https://review.opendev.org/696992
17:40:42 openstackgerrit Merged openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072
17:40:51 openstackgerrit Merged openstack/nova master: Extend NeutronFixture to handle multiple bindings https://review.opendev.org/696246
17:40:58 openstackgerrit Merged openstack/nova master: Do not mock setup net and migrate inst in NeutronFixture https://review.opendev.org/696247
18:22:23 efried mriedem: are you okay with stephenfin's update to at doc patch? Since you both have hands in it, if you're okay with it I'll fast approve, taking stephenfin's authorship as implicit approval and since it's docs...
18:22:41 melwitt johnthetubaguy: hey, are you around bychance?
18:37:12 mriedem efried: yet to look at it
18:37:29 mriedem but soon, very soon....muwahhaaha
18:40:02 openstackgerrit Matt Riedemann proposed openstack/nova master: Add cross-cell resize tests for _poll_unconfirmed_resizes https://review.opendev.org/698322
18:40:03 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: debug cross-cell resize https://review.opendev.org/698304
18:41:25 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: debug cross-cell resize https://review.opendev.org/698304
18:44:19 mriedem efried: stephenfin's changes look fine to me
18:45:00 efried +A
18:45:26 mriedem if someone can push through https://review.opendev.org/#/c/696509/ it's the current bottom of the nova-net removal series; it looks like the rest of the series after that is now in merge conflict so the whole thing has to be rebased.
18:52:46 efried mriedem: I'm gonna try to hit that today, but it keeps getting pushed down my stack :(
18:53:39 mriedem ack it's pretty mechanical so anyone should be able to hit it
19:02:40 openstack bugzilla.redhat.com bug 1688838 in openstack-nova "Ironic should not treat cpu_arch as mandatory" [Medium,New] - Assigned to mwitt
19:02:40 melwitt TheJulia: do you know whether cpu_arch is supposed to be required from an ironic pov? https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L103-L106 is it valid for a deployment *not* to specify cpu_arch? (for example, in a single arch environment) context is this bug https://bugzilla.redhat.com/show_bug.cgi?id=1688838
19:06:12 sean-k-mooney gibi: by the way i assume attaching a port with a resocue request to a existing instance is still supported correct. it was declared out of scope in stien and it was not mentioned as adressed in train. is that on your ussuri todo list?
19:08:19 sean-k-mooney i assume we would have updated https://github.com/openstack/nova/blob/master/releasenotes/notes/reject-interface-attach-with-port-resource-request-17473ddc5a989a2a.yaml if it was supported or added another release note
19:18:41 melwitt jroll: ^ maybe you might know (ironic driver question from me)
19:31:46 mriedem sean-k-mooney: not supported - the request has to be validated with placement on attach and that isn't done
19:32:11 sean-k-mooney ya that is what i understood too
19:32:27 sean-k-mooney im doing a downstream docs review and wanted to make sure that was called out
19:33:13 openstack Launchpad bug 1835637 in OpenStack Compute (nova) "(404) NOT_FOUND - failed to perform operation on queue 'notifications.info' in vhost '/nova' due to timeout" [Undecided,Incomplete]
19:33:13 mnaser eandersson: is this similar to what you've been running into? https://bugs.launchpad.net/nova/+bug/1835637
20:04:59 openstackgerrit Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for Cyborg access https://review.opendev.org/631242
20:05:00 openstackgerrit Sundar Nadathur proposed openstack/nova master: Define Cyborg ARQ binding notification event. https://review.opendev.org/692707
20:05:00 openstackgerrit Sundar Nadathur proposed openstack/nova master: Add Cyborg device profile groups to request spec. https://review.opendev.org/631243
20:05:01 openstackgerrit Sundar Nadathur proposed openstack/nova master: Compose accelerator PCI devices into domain XML in libvirt driver. https://review.opendev.org/631245
20:05:01 openstackgerrit Sundar Nadathur proposed openstack/nova master: Create and bind Cyborg ARQs. https://review.opendev.org/631244
20:05:02 openstackgerrit Sundar Nadathur proposed openstack/nova master: Enable hard reboot with accelerators. https://review.opendev.org/697940
20:05:02 openstackgerrit Sundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted. https://review.opendev.org/673735
20:05:03 openstackgerrit Sundar Nadathur proposed openstack/nova master: Pass accelerator requests to each virt driver from compute manager. https://review.opendev.org/698581
20:05:03 openstackgerrit Sundar Nadathur proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999
20:07:07 openstackgerrit Sundar Nadathur proposed openstack/nova master: Create and bind Cyborg ARQs. https://review.opendev.org/631244
20:07:08 openstackgerrit Sundar Nadathur proposed openstack/nova master: Compose accelerator PCI devices into domain XML in libvirt driver. https://review.opendev.org/631245
20:07:08 openstackgerrit Sundar Nadathur proposed openstack/nova master: Pass accelerator requests to each virt driver from compute manager. https://review.opendev.org/698581
20:07:09 openstackgerrit Sundar Nadathur proposed openstack/nova master: Enable hard reboot with accelerators. https://review.opendev.org/697940
20:07:09 openstackgerrit Sundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted. https://review.opendev.org/673735
20:07:10 openstackgerrit Sundar Nadathur proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999
20:44:16 openstackgerrit Lee Yarwood proposed openstack/nova stable/stein: block_device: Copy original volume_type when missing for snapshot based volumes https://review.opendev.org/696686
20:44:46 openstackgerrit Lee Yarwood proposed openstack/nova stable/rocky: block_device: Copy original volume_type when missing for snapshot based volumes https://review.opendev.org/697260
20:45:03 openstackgerrit Lee Yarwood proposed openstack/nova stable/queens: block_device: Copy original volume_type when missing for snapshot based volumes https://review.opendev.org/697261
20:49:06 eandersson mnaser I dont't think it's the same issue, but could be similar
20:49:28 mnaser eandersson: seems like the queues just crash even on a restart
20:49:34 eandersson https://github.com/rabbitmq/rabbitmq-server/issues/641
20:49:41 mnaser even with 3.8.1
20:49:44 eandersson This is the issue we are running into
20:50:19 mnaser given that we have k8s running side by side our entire infra now, we're just probably going to run one single non-clustered rabbitmq instance
20:50:23 eandersson but yea could be the same because we see different symptons each time
20:50:26 mnaser for every service
20:51:53 eandersson Yea - this has been very draining for us
20:52:06 eandersson Some of our RabbitMQ clusters has 17k+ queues as well
20:52:27 eandersson Since every compute ends up with like at least 7 queues
20:52:52 eandersson btw mnaser do you have network partition auto-healing enabled?
20:53:11 eandersson or pause_minority rather
20:54:22 eandersson I believe the trigger for these issues is that when RabbitMQ comes back up it is starting to accept connections before it is fully recovered.
20:54:45 eandersson At least in 3.6.X we had that issue. Could be a new issue in 3.7.X
20:55:35 mnaser eandersson: i ran into it even with 3.8.1
20:55:56 mnaser eandersson: cluster_partition_handling, pause_minority
20:56:45 eandersson I tried to tell the RabbitMQ guys about this, but I don't know how to reproduce it properly.
20:56:58 eandersson Plus I run like 3.7.5 so they just keep telling me to upgrade.
20:57:46 eandersson But after the RabbitMQ 3.6.3 debacle we don't upgrade too frequently without first testing it properly. So takes time.
21:00:05 Sundar dansmith: Would it help to discuss in IRC and then summarize in Gerrit?
21:01:14 dansmith Sundar: if you were ever in irc, sure
21:01:56 dansmith Sundar: what I want is discussion instead of replies of "no you're wrong" two minutes before pushing up replacement set without addressing the thing
21:02:45 Sundar dansmith Sure. I addressed most of your points, BTW.
21:03:12 Sundar I think the disconnect is the understanding of the object model. Please see https://review.opendev.org/#/c/631243/46/nova/accelerator/cyborg.py@26
21:04:31 dansmith Sundar: I'm not sure what your point is
21:05:03 dansmith Sundar: I will eventually be able to have one crypto accelerator and one gzip accelerator, right?
21:05:20 Sundar dansmith: Yes, for the same instance
21:05:30 dansmith and those are two device profiles?
21:05:42 Sundar No, one device profile with 2 request groups
21:05:54 dansmith okay, so an instance will only ever have one device profile?
21:06:10 Sundar Yes. That single device profile's name is set in the flavor.
21:06:25 dansmith so, why are you setting the tag in the event to the device profile?
21:07:24 Sundar That was the only logical choice for a tag that seemed relevant.
21:07:59 dansmith setting the tag means that the event needs to be multiplexed for the instance
21:08:22 dansmith which is why we set it to the port id for neutron ports, for example, because there are multiple ports per instance
21:08:54 dansmith I'm not sure why it was decided the encompass all of the accelerators for an instance into a single entity, but alas
21:09:29 Sundar Would we ever need multiple events per instance? For example, for hot adds/deletes in the future, by updating the device profile?
21:09:47 dansmith Sundar: those would be different event types
21:10:47 Sundar Then what exactly is the problem -- that the tag is superfluous?
21:11:08 dansmith setting the tag implies that there can be multiples, so ... yes
21:12:01 Sundar dansmith: Got the disconnect. We don;t use the tag, at least not today. With hot adds/deletes, since it it is going to be another event type, we still don;t need it.
21:12:38 Sundar dansmith: BTW, the idea of one device profile per instance came from the idea of setting one device profile name in the instance.
21:12:42 dansmith I dunno what those would look like, but I would hope that there is some indication of what device the "thing got added" event pertains to
21:13:16 dansmith ...which would be done with a tag
21:14:44 Sundar dansmith: It may make sense to have a single notification for an update too -- because there is not much that Nova can with a partial update knowing that the next event may indicate a failure and things need to be rolled back
21:15:00 Sundar That is the same reasoning as for the bind logic here

Earlier   Later