| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-01 | |||
| 16:00:25 | artom | Why is it trying to pickle the session? o_O | |
| 16:00:42 | mriedem | yeah, it pickles everything in InstanceMetadata, which includes a bunch of versioned objects | |
| 16:00:44 | mriedem | like Instance | |
| 16:01:04 | mriedem | and since we do db api stuff in the objects now, i'm not surprised Session is pulled in | |
| 16:01:14 | artom | Hrmm | |
| 16:01:20 | artom | This is with memcache | |
| 16:01:25 | mriedem | i thought at some point someone from red hat was working on fixing this up because of performance/security issues with what we were serializing | |
| 16:01:31 | artom | Which I suspect isn't tested in the gate anywhere | |
| 16:01:47 | artom | Not to my knowledge | |
| 16:01:56 | artom | But I'm not omniscient ;) | |
| 16:03:05 | mriedem | we don't use memcached in the gate as far as i know, default dogpile cache is in-memory dict i think | |
| 16:03:20 | artom | Makes sense, which is why we never noticed this | |
| 16:03:28 | mriedem | there is a new etcd backend for oslo.cache, and etcd is installed in jobs on master now in devstack, on the primary node | |
| 16:03:43 | mriedem | so we could start testing with that.... | |
| 16:04:03 | clarkb | mriedem: I think we do install memcached at least | |
| 16:04:18 | clarkb | so you can strt testing with either or | |
| 16:04:19 | mriedem | ok we just don't configure to use it | |
| 16:04:20 | artom | Would it hit the issue though? I thought the issue is that python-memcache pickles the stuff we hand it | |
| 16:04:34 | mriedem | kolla is running with memcache configured for this, | |
| 16:04:37 | mriedem | that's how they hit the bug | |
| 16:04:45 | mriedem | http://logs.openstack.org/73/469373/1/check/gate-kolla-ansible-dsvm-deploy-centos-source-centos-7-nv/8cecb36/logs/kolla/nova/nova-api.log | |
| 16:05:00 | artom | So either we dramatically simplify InstanceMetadata so that it's picklable | |
| 16:05:27 | artom | Or... can python-memcache not pickle stuff? I guess it has to to put them in RAM | |
| 16:05:44 | mriedem | it uses pickle | |
| 16:05:48 | mriedem | i don't see any option not to | |
| 16:05:52 | mriedem | in python-memached i mean | |
| 16:05:57 | mriedem | i was looking at the code earlier | |
| 16:05:58 | artom | What if we just IntanceMetada.to_primitive()? | |
| 16:06:04 | artom | And then from_primitive() when we get it back? | |
| 16:06:07 | mriedem | i thought about something like that | |
| 16:06:26 | mriedem | since we have that for the versioned objects contained within | |
| 16:06:28 | artom | I sense a but coming | |
| 16:06:33 | mriedem | no but | |
| 16:06:38 | mriedem | would just need to hack on it and test it out | |
| 16:07:22 | artom | I *think* I could grab that? I kinda have other stuff going on downstream, but I could try to make it work | |
| 16:09:03 | efried | mriedem Re powervm blueprint: we need to keep it open at least long enough to get the docs changes in. We still need to have an internal powwow as to whether we want to drive more function this release; will get back to you soon on that. | |
| 16:12:51 | mriedem | artom: ack, not laying it on you, was just wondering if you had heard about this or knew someone working on it already | |
| 16:12:54 | mriedem | but don't feel obliged | |
| 16:13:10 | artom | mriedem, my parents raised me on guilt, I feel obliged ;) | |
| 16:13:37 | mriedem | that works for me | |
| 16:13:50 | artom | mriedem, in seriousness, it's a bug, I have the solution in my head, it's the responsible thing to do | |
| 16:14:14 | artom | Plus, how hard can it be? | |
| 16:58:31 | mriedem | sean-k-mooney: don't you have to configure nova.conf for os-vif plugins that are out of tree? | |
| 16:58:41 | mriedem | or does os-vif just automagically register those if it can import them via stevedore? | |
| 17:03:40 | mriedem | this would be their extension point https://github.com/Juniper/contrail-nova-vif-driver/blob/master/setup.py#L27 | |
| 17:03:46 | mriedem | which i guess needs to be renamed also | |
| 17:05:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Connecting Nova to DRBD storage nodes directly https://review.openstack.org/256292 | |
| 17:09:02 | mriedem | ^ is an easy bp win for something that's been sitting around nearly 2.5 years | |
| 17:09:10 | mriedem | i finally just cleaned it up myself | |
| 17:13:22 | openstackgerrit | Michael Henkel proposed openstack/nova master: contrail: add vrouter VIF plugin type support https://review.openstack.org/441183 | |
| 17:14:15 | sean-k-mooney | mriedem: reading back | |
| 17:14:36 | openstackgerrit | John Griffith proposed openstack/nova master: WIP/POC Implement new attach Cinder flow https://review.openstack.org/330285 | |
| 17:15:12 | sean-k-mooney | mriedem: ya os-vif finds the plugins by stevadore so if they update the plugin name they also need to update there setup.cfg | |
| 17:18:21 | openstackgerrit | sean mooney proposed openstack/os-vif master: set mtu on all code paths https://review.openstack.org/469951 | |
| 17:20:07 | sean-k-mooney | mriedem: drdb is https://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device ? | |
| 17:24:13 | mriedem | sean-k-mooney: yeah | |
| 17:27:11 | sean-k-mooney | mriedem: hum look interesting. looking at the patch all the hard work of actully mounting the volume is done by libvirt so ya that looks like an easy win | |
| 17:27:51 | sean-k-mooney | mriedem: is there a minium libivrt version that it works with that should be in the release note? | |
| 17:28:27 | mriedem | no the work is done in os-brick | |
| 17:29:55 | sean-k-mooney | mriedem: oh yes connector is from os_brick.initiator not the libvirt dirver | |
| 17:47:33 | cfriesen_ | Am I correct that nova commit 17a8e8 (Block flavor creation until main database is empty) in Newton basically means that on an upgrade from Mitaka to Newton you can't create flavors until you run "nova-manage db online_data_migrations"? | |
| 17:49:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: fix call args to destroy() during live migration rollback https://review.openstack.org/469958 | |
| 17:50:46 | mriedem | cfriesen_: yes | |
| 17:50:51 | mriedem | flavors are migrated to the api db | |
| 17:51:48 | cfriesen_ | mriedem: I don't see that mentioned anywhere in the Newton upgrade notes...I thought that normally we didn't require "nova-manage db online_data_migrations" to be run immediately after an upgrade, only sometime before the next upgrade. | |
| 17:53:28 | mriedem | it would have been mitaka | |
| 17:53:37 | mriedem | it's a blocker to get to newton | |
| 17:53:39 | mriedem | you run it during mitaka | |
| 17:54:07 | cfriesen_ | the online_data_migrations in mitaka don't touch flavors | |
| 17:54:15 | cfriesen_ | only in newton | |
| 17:54:26 | cfriesen_ | i think it *should* have been in mitaka | |
| 17:54:35 | mriedem | you're going to make me git blame this aren't you | |
| 17:54:44 | cfriesen_ | it was dansmith | |
| 17:54:54 | mriedem | yes i know | |
| 17:55:00 | mriedem | that's not why i'd git blame | |
| 17:55:05 | mriedem | i'd blame dan regardless :) | |
| 17:55:10 | cfriesen_ | lol | |
| 17:55:38 | dansmith | cfriesen_: no, it's not "just any time" it's "after things are back online" | |
| 17:55:44 | mriedem | https://review.openstack.org/#/c/294771/ | |
| 17:55:50 | jaypipes | melwitt: would you like reviews on the remainder of your quota series or reviews on your add project/user to placement series? pick one :) | |
| 17:56:06 | dansmith | that's the gate for a lot of compatibility things, so you're likely taking extra perf penalties until you've completed all of those things | |
| 17:56:56 | mriedem | which is the blocker migration? | |
| 17:57:16 | mriedem | 291 right? https://github.com/openstack/nova/commit/bf8b3946e6964059dce581f54ed4f928e2cdf5ab | |
| 17:57:23 | mriedem | oh no | |
| 17:58:33 | cfriesen_ | dansmith: I get that there would be performance penalties, but I thought nova tried to actually do online conversions as needed if you hadn't run it yet. | |
| 17:58:59 | cfriesen_ | dansmith: it appears that for flavors nova just didn't do the online conversions and relied on the DB to be converted | |
| 17:59:00 | dansmith | it depends on what it is | |
| 17:59:09 | dansmith | not for flavor moving across dbs | |
| 17:59:59 | cfriesen_ | the newton upgrade docs just say "The newton release has a lot of online migrations that must be performed before you will be able to upgrade to ocata. Please take extra note of this fact and budget time to run these online migrations before you plan to upgrade to ocata." | |
| 18:00:19 | cfriesen_ | that note should probably say that you need to run it right after upgrading to newton | |
| 18:00:28 | cfriesen_ | otherwise you won't be able to create flavors | |
| 18:01:14 | dansmith | attributes. Until this is complete, new attempts to create flavors | |
| 18:01:14 | dansmith | in your main database to the API database, retaining all | |
| 18:01:14 | dansmith | release, the online data migrations will move any flavors you have | |
| 18:01:14 | dansmith | Flavors are being moved to the API database for CellsV2. In this | |
| 18:01:15 | dansmith | will return an HTTP 409 to avoid creating flavors in one place that | |
| 18:01:16 | dansmith | may conflict with flavors you already have and are yet to be | |
| 18:01:17 | dansmith | migrated. | |
| 18:01:18 | dansmith | ^^ | |
| 18:02:14 | cfriesen_ | dansmith: I don't see that at https://docs.openstack.org/releasenotes/nova/newton.html#upgrade-notes | |