| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-02 | |||
| 14:29:17 | lyarwood | aspiers: yeah there's only LVM support for that at the moment | |
| 14:29:23 | aspiers | lyarwood: that's fine | |
| 14:29:35 | aspiers | enable_plugin barbican | |
| 14:29:37 | aspiers | NOVA_BACKEND=LVM | |
| 14:29:47 | aspiers | is that enough, combined with the extra config options? | |
| 14:29:53 | openstackgerrit | Merged openstack/nova stable/stein: Fix AttributeError in RT._update_usage_from_migration https://review.opendev.org/673856 | |
| 14:29:58 | aspiers | [[post-config|$NOVA_CONF]] | |
| 14:30:00 | aspiers | [ephemeral_storage_encryption] | |
| 14:30:02 | aspiers | key_size = 256 | |
| 14:30:04 | aspiers | cipher = aes-xts-plain64 | |
| 14:30:06 | aspiers | enabled = True | |
| 14:30:08 | aspiers | lyarwood: something like that? | |
| 14:30:22 | lyarwood | aspiers: yeah something like that | |
| 14:30:28 | lyarwood | aspiers: I haven't played around with it in a while | |
| 14:30:31 | aspiers | OK cool thanks I'll give it a go! | |
| 14:32:19 | aspiers | ah I need the git URL after enable_plugin barbican | |
| 14:58:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-osprofiler-redis job to experimental queue https://review.opendev.org/674326 | |
| 15:04:57 | mriedem | you could look at the barbican zuul ci job setup for some of that | |
| 15:05:26 | mriedem | they also have https://github.com/openstack/barbican/blob/master/devstack/local.conf.example | |
| 15:05:29 | mriedem | not sure how accurate it is | |
| 15:06:07 | mriedem | else https://github.com/openstack/barbican/blob/master/playbooks/legacy/barbican-devstack-base/run.yaml | |
| 15:32:32 | openstackgerrit | Merged openstack/nova master: trivial: Remove unused function parameter https://review.opendev.org/671796 | |
| 15:59:31 | spatel | sean-k-mooney: hey!! | |
| 16:00:28 | spatel | i found issue and it was very stupid :( somehow my kickstart file got messed up and it created /var filesystem with only 150G, that is why i was not able to create more instance. | |
| 16:13:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Convert nova-lvm job to zuul v3 and modernize https://review.opendev.org/674345 | |
| 16:20:16 | openstackgerrit | Merged openstack/nova master: Add method 'all_required_traits' to scheduler utils https://review.opendev.org/670297 | |
| 16:48:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Don't mock something globally https://review.opendev.org/674351 | |
| 17:10:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Convert nova-lvm job to zuul v3 https://review.opendev.org/674345 | |
| 17:13:47 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Modernize nova-lvm job https://review.opendev.org/674356 | |
| 17:30:23 | aspiers | lyarwood: looks like a reboot loses devstack's /dev/loopN which are needed for the LVM volumes and devstack doesn't configure auto-setup on boot :-/ | |
| 17:31:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-osprofiler-redis job to experimental queue https://review.opendev.org/674326 | |
| 17:38:38 | openstackgerrit | Merged openstack/nova master: doc: correct the information of 'cpu_map' https://review.opendev.org/673272 | |
| 18:00:40 | eandersson | Anyone seen this error when cold migrating (Rocky) http://paste.openstack.org/show/755342/ | |
| 18:00:46 | eandersson | > ValueError: Circular reference detected | |
| 18:01:12 | openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | |
| 18:01:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | |
| 18:01:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support translating CPU policy extra specs, image meta https://review.opendev.org/671801 | |
| 18:01:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell' https://review.opendev.org/674395 | |
| 18:01:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add proxy fields to NUMATopology object https://review.opendev.org/674104 | |
| 18:01:15 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Remove unnecessary argument https://review.opendev.org/674396 | |
| 18:09:58 | mriedem | efried: looks like we need a warnings filter to avoid blowing up the console logs in functional test runs https://logs.opendev.org/61/673961/1/gate/nova-tox-functional-py36/a4cb2af/job-output.txt.gz#_2019-08-01_18_05_40_322077 | |
| 18:10:15 | mriedem | b'/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/openstack/service_description.py:224: UnsupportedServiceVersion: Service placement has no discoverable version. The resulting Proxy object will only have direct passthrough REST capabilities.' | |
| 18:10:22 | mriedem | getting all sorts of warnings from openstacksdk | |
| 18:10:46 | mriedem | that or we actually have something to fix | |
| 18:13:55 | mriedem | i'll push a patch to the sdk to fix this one https://github.com/openstack/openstacksdk/blob/master/openstack/config/cloud_region.py#L173 | |
| 18:18:53 | openstack | Launchpad bug 1838807 in OpenStack Compute (nova) "UnsupportedServiceVersion: Service placement has no discoverable version. The resulting Proxy object will only have direct passthrough REST capabilities." [Medium,Confirmed] | |
| 18:18:53 | mriedem | https://bugs.launchpad.net/nova/+bug/1838807 | |
| 18:37:47 | eandersson | mriedem, you seen this before? http://paste.openstack.org/show/755342/ | |
| 18:39:31 | mriedem | hmm, i've seen something like that in the past related to versioned notification payloads but that was fixed a long time ago | |
| 18:40:23 | mriedem | https://github.com/openstack/nova/commit/3bf177a59cfd0b4e74dba256c3466ba2ea9bfbf7#diff-f009347572c3f7457d296a3b652002b5 | |
| 18:40:59 | eandersson | We for sure have that patch - this is stable/rocky (~5 commits behind) | |
| 18:40:59 | mriedem | so i wonder if there is a context in whatever you're seeing get serialized and that's blowing things up | |
| 18:41:18 | mriedem | actually hmm | |
| 18:41:19 | mriedem | legacy_spec = jsonutils.loads(jsonutils.dumps(legacy_spec)) | |
| 18:41:44 | mriedem | i think there is a comment on that line about needing to do that to avoid a valuerror | |
| 18:42:02 | mriedem | https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/migrate.py#L291 | |
| 18:42:11 | eandersson | > legacy_props.pop('context', None) | |
| 18:42:41 | eandersson | maybe we need to do the same on both? | |
| 18:42:55 | mriedem | legacy_spec should be a dict form of the RequestSpec object w/o a context in it | |
| 18:43:06 | mriedem | the dance done there is b/c of datetime fields afaict | |
| 18:43:13 | mriedem | er per the comment from bauzas | |
| 18:43:24 | mriedem | i also recently investigated this b/c i wanted to remove that... | |
| 18:44:37 | mriedem | yeah https://review.opendev.org/#/c/582417/ | |
| 18:46:00 | eandersson | Would that be possible to backport? | |
| 18:46:10 | mriedem | no it has an rpc change | |
| 18:46:17 | mriedem | but i'm not sure it would help you anyway | |
| 18:46:43 | mriedem | note that we're still serializing/deserializing the ImageMeta object https://review.opendev.org/#/c/582417/7/nova/compute/rpcapi.py@741 | |
| 18:47:38 | mriedem | i think, but don't know why, that is only in here for unit/functional tests that rely on the oslo.messaging rpc fake driver, but it doesn't make much sense to me because our functional tests pass things with datetimes over rpc all the time, | |
| 18:47:48 | mriedem | like passing RequestSpec to the scheduler's select_destinations call | |
| 18:49:01 | mriedem | do you have any out of tree changes to RequestSpec? | |
| 18:49:16 | eandersson | Not that I am aware of | |
| 18:49:29 | eandersson | The only changes we do have are customer scheduler filters and weighers | |
| 18:49:29 | mriedem | is this an isolated incident? | |
| 18:49:43 | eandersson | isolated so far yea | |
| 18:49:51 | eandersson | but we haven't done a lot of migrations | |
| 18:50:16 | eandersson | and this happened during like test run 3 | |
| 18:50:34 | mriedem | hmm, ok so this is a cold migratoin (or resize) that is failing on one host and trying to reschedule to another and then failing when trying to serialize the legacy request spec b/c rpc casting to prep_resize on the alternative host | |
| 18:50:37 | eandersson | *customer not customer | |
| 18:50:43 | mriedem | the difference is probably in the reschedule | |
| 18:50:50 | eandersson | *custom | |
| 18:50:51 | eandersson | man | |
| 18:53:07 | mriedem | i'll poke through that flow, we don't really hit migration reschedules in tempest integration runs unless it's in a multinode job | |
| 18:53:12 | mriedem | and not intentionally | |
| 18:53:30 | eandersson | Does it use placement for scheduling migrations? | |
| 18:54:17 | eandersson | Due to how this fleet is set up it is a little too likely to hit Rescheduling errors at the moment in general. | |
| 18:55:15 | eandersson | We heavily favor stacking VMs, plus Placement isn't properly configured, as we still rely on Aggregates for over-commit etc. | |
| 18:56:01 | eandersson | One of the downsides with doing a Mitaka to Rocky jump is that you miss things in releasenotes. :p | |
| 18:56:19 | mriedem | placement is used for scheduling, | |
| 18:56:49 | mriedem | so yes, the scheduler is going to ask placement for candidate hosts and the scheduler will pick a primary host after weighing and some alternates | |
| 18:57:01 | mriedem | and those alternates are what get passed between conductor and compute during reschedules | |
| 18:57:21 | mriedem | so after the initial scheduling request placement isn't really in the picture except for cleaning up allocations once the resize is confirmed/reverted | |
| 18:57:43 | mriedem | i mean, i don't know what is triggering your reschedule - that should be clear from the compute logs, | |
| 18:57:50 | eandersson | I have a good idea. | |
| 18:57:59 | mriedem | but the valueerror thing is from something with the RequestSpec which doesn't really have anything to do with placement | |
| 19:00:17 | mriedem | we have functional tests that trigger resize reschedules but they use fake rpc drivers and serialization fixtures for the context so those might be masking something, | |
| 19:00:30 | mriedem | i'd have to stand up a 2 node devstack and hack it up to recreate | |
| 19:01:07 | mriedem | if you could recreate, it would be good to dump that thing that gets serialized before it blows up | |
| 19:01:17 | eandersson | Yep - I'll see if we can catch it | |
| 19:01:22 | eandersson | Maybe even in the lab. | |
| 19:01:44 | mriedem | i'm not sure if i can get something setup to recreate in the next hour | |